vSphere 4 Update 2 Released

VMware vSphere 4, Update 2 has been released with the following changes to ESXi:

The following information provides highlights of some of the enhancements available in this release of VMware ESXi:

• Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.


• Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.


• Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.


• Enhancement of the resxtop utility— vSphere 4.0 U2 includes an enhancement of the performance monitoring utility, resxtop. The resxtop utility now provides visibility into the performance of NFS datastores in that it displays the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s (guest latency).


• Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04. For a complete list of supported guest operating systems with this release, see the VMware Compatibility Guide.

Resolved Issues – In addition, this release delivers a number of bug fixes that have been documented in theResolved Issues section.

ESXi 4 Update 2 Release Notes

Noted in the release is the official support for AMD's IOMMU in Opteron 6100 and 4100 processors - available in 1P, 2P and 4P configurations. This finally closes the (functional) gap between AMD Opteron and Intel's Nehalem line-up. Likewise, FT support for many new Intel processors has been added. Also, the addition of NFS performance counters in ESXTop will make storage troubleshooting a bit easier. Grab you applicable update at VMware's download site now (SnS required.)

Quick Take: IBM Tops VMmark, Crushes Record with 4P Nehalem-EX

It was merely a matter of time before one of the new core-rich titans - the Intel's 8-core "Beckton" Nehalem-EX (Xeon 7500) or AMD's 12-core "Magny-Cours" (Opteron 6100) - was to make a name for itself on VMware's VMmark benchmark. Today, Intel draws first blood in the form of an 4-processor, 32-core, 64-thread, monster from IBM: the x3850 X5 running four Xeon X7560 (2.266GHz - 2.67GHz w/turbo, 130W TDP, each) and 384GB of DDR3-1066 low-power registered DIMMs. Weighing-in at 70.78@48 tiles, the 4P IBM System x3850 handily beats the next highest system - the 48-core DL785 G5 which set the record of 53.73@35 tiles back in August, 2009 - and bests it by over 30%.

At $3,800+ per socket for the tested Beckton chip, this is no real 2P alternative. In fact, a pair of Cisco UCS B250 M2 blades will get 52 tiles running for much less money. Looking at processor and memory configurations alone, this is a $67K+ enterprise server, resulting in a moderately-high $232/VM price point for the IBM x3850 X5.

SOLORI's Take: The most interesting aspect of the EX benchmark is its clock-adjusted scaling factor: between 70% and 91% versus a 2P/8-core Nehalem-EP reference (Cisco UCS, B200 M1, 25.06@17 tiles). The unpredictable nature of Intel's "turbo" feature - varying with thermal loads and per-core conditions - makes an exact clock-for-clock comparison difficult. However, if the scaling factor is 90%, the EX blows away our previous expectations about the platform's scalability. Where did we go wrong when we predicted a conservative 44@39 tiles? We're looking at three things: (1) a bad assumption about the effectiveness of "turbo" in the EP VMmark case (setting Ref_EP_Clock to 3.33 GHz), and (2) underestimating EX's scaling efficiency (assumed 70%), (3) assuming a 2.26GHz clock for EX.

Chosing our minimum QPI/HT3 scalability factor of 75%, the predicted performance was derived this way from HP Proliant BL490 G6 as a baseline:

Est. Tiles = EP_Tiles_per_core( 2.13 ) * 32 cores * Scaling_Efficiency( 75% ) * EX_Clock( 2.26 ) / EP_Clock( 2.93 ) = 39 tiles

Est. Score = Est_Tiles( 40 ) * EP_Score_per_Tile( 1.43 ) * Est_EX_Clock( 2.26 ) / Ref_EP_Clock( 2.93 ) = 44.12

Est. Nehalem-EX VMmark -> 44.12@39 tiles

Correcting for the as-tested clock/turbo numbers, and using AMD's 2P-to-4P VMmark scaling efficiency of 83%, and shifting to the new UCS baseline (with newer ESX version) the Nehalem-EX prediction factors to:

Est. Tiles = EP_Tiles_per_core( 2.13 ) * 32 cores * Scaling_Efficiency( 83% ) * EX_Clock( 2.67 ) / EP_Clock( 2.93 ) = 51 tiles

Est. Score = Est_Tiles( 51 ) * EP_Score_per_Tile( 1.47 ) * Est_EX_Clock( 2.67 ) / Ref_EP_Clock( 2.93 ) = 68.32

Est. Nehalem-EX VMmark -> 68.3@51 tiles

Clearly, this approach either overestimates the scaling efficiency or underestimates the "turbo" mode. IBM claims that a 2.93 GHz "turbo" setting is viable where Intel suggests 2.67 GHz is the maximum, so there is a source of potential bias. Looking at the tiles-per-core ratio of the VMmark result, the Nehalem-EX drops from 2.13 tiles per core on EP/2P platforms to 1.5 tiles per core on EX/4P platforms - about a 30% drop in per-core loading efficiency. That indicator matches well with our initial 75% scaling efficiency moving from 2P to 4P - something that AMD demonstrated with Istanbul last August. Given the high TDP of EX and IBM's 2.93 GHz "turbo" specification, it's possible that "turbo" is adding clock cycles (and power consumption) and compensating for a "lower" scaling efficiency than we've assumed. Looking at the same estimation with 2.93GHz "clock" and 71% efficiency (1.5/2.13), the numbers fall in line with VMmark:

Est. Tiles = EP_Tiles_per_core( 2.13 ) * 32 cores * Scaling_Efficiency( 71% ) * EX_Clock( 2.93 ) / EP_Clock( 2.93 ) = 48 tiles

Est. Score = Est_Tiles( 48 ) * EP_Score_per_Tile( 1.47 ) * Est_EX_Clock( 2.93 ) / Ref_EP_Clock( 2.93 ) = 70.56

Est. Nehalem-EX VMmark -> 70.56@48 tiles

This give us a good basis for evaluating 2P vs. 4P Nehalem systems: scaling factor of 71% and capable of pushing clock towards the 3GHz mark within its thermal envelope. Both of these conclusions fit typical 2P-to-4P norms and Intel's process history.

SOLORI's 2nd Take: So where does that leave AMD's newest 12-core chip? To date, no VMmark exists for AMD's Magny-Cours, and AMD chips tend not to do as well in VMmark as their Intel peers do to the benchmarks SMT-friendly loads. However, we can't resist using the same analysis against AMD/MC's 2.4GHz Opteron 6174SE (theoretical) using the 2P HP DL385 G6 as a baseline for core loading and the HP DL785 G6 for tile performance (best of the best cases):

Est. Tiles = HP_Tiles_per_core( 0.92 ) * 48 cores * Scaling_Efficiency( 83% ) * MC_Clock( 2.3 ) / HP_Clock( 2.6 ) = 33 tiles

Est. Score = Est_Tiles( 34 ) * HP_Score_per_Tile( 1.54 ) * Est_MC_Clock( 2.3 ) / Ref_HP_Clock( 2.8 ) = 41.8

Est. 4P Magny-Cours VMmark -> 41.8@33 tiles

That's nowhere near good enough to top the current 8P, 48-core Istanbul VMmark at 53.73@35 tiles, so we'll likely have to wait for faster 6100 parts to see any new AMD records. However, assuming AMD's proposition is still "value 4P" so about 200 VM's at under $18K/server gets you around $90/VM or less.

Quick Take: Q1 DRAM Price Follow-up, 8GB DDR3 Below Target

In September 2009 we predicted that average 8GB DIMM prices (DDR2 and DDR3) would reach $565/stick by year end (with DDR3 being higher than DDR2) and at now we're seeing the reversal of fortunes for DDR2. At year end, the average price for benchmark DDR2/DDR3 was $591 retail, with promotional pricing pushing that below$550 as predicted. Today, we're seeing DDR3 begin to overtake DDR2 in the 8GB ECC category, dropping below $510/stick, while DDR2 climbs to $550/stick (promotional, on $625/stick retail.)

In 4GB ECC configurations, DDR2 enjoys only a slight retail advantage (13%) while promotional pricing (likely due to inventory reduction initiatives) are providing a bit better value short term. However, the price gap is only 1/2 the power gap, with DDR3 delivering a greater than 35% reduction in power over its DDR2 equivalent (about $1.25/year/stick at $0.10/kWh). The honeymoon is almost over for DDR2.

Benchmark Server (Spot) Memory Pricing - Dual Rank DDR2 Only
DDR2 Reg. ECC Series (1.8V)	Price Jun '09	Price Sep '09	Price Dec '09	Price Mar '10
KVR800D2D4P6/4G 4GB 800MHz DDR2 ECC Reg with Parity CL6 DIMM Dual Rank, x4 (5.400W operating)	$100.00	$117.00 up 17%	$140.70 up 23% promo	$128.90 ($151 retail)
KVR667D2D4P5/4G 4GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (5.940W operating)	$80.00	$103.00 up 29%	$97.99 down 5%	$128.74 ($149 retail)
KVR667D2D4P5/8G 8GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (7.236W operating)	$396.00	$433.00	$433.00 (promo)	$550.00 (Promo price, retail $625)

Benchmark Server (Spot) Memory Pricing - Dual Rank DDR3 Only
DDR3 Reg. ECC Series (1.5V)	Price Jun '09	Price Sep '09	Price Dec '09	Price Mar '10
KVR1333D3D4R9S/4G 4GB 1333MHz DDR3 ECC Reg w/Parity CL9 DIMM Dual Rank, x4 w/Therm Sen (3.960W operating)	$138.00	$151.00 up 10%	$135.99 down 10%	$150.74 ($170 retail)
KVR1066D3D4R7S/4G 4GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (5.085W operating)	$132.00	$151.00 up 15%	$137.59 down 9% (promo)	$150.74 ($170 retail)
KVR1066D3D4R7S/8G 8GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (4.110W operating)	$1035.00	$917.00 down 11.5%	$667.00 down 28%	$506.59 (retail $584, avail. 3/15)
KVR1333D3D4R9S/8GHA 8GB 1333MHz DDR3 ECC Reg CL9 DIMM 2R x4 w/TS Server Hynix A (4.635W operating)				$584.00

SOLORI's Take: With strong DDR3 demand and short-falls in DDR2 supply (according to DRAMeXchange), the only thing keeping DDR3 prices above DDR2 at this point is demand and inventory. As Q2/2010 introduces a rush of new workstation and server products based on DDR3 systems, the DRAM production ramp will eventually stabilize demand somewhere towards the end of Q3/2010. Meanwhile, technology companies like VMware, Microsoft, Intel and AMD are betting on new infrastructure spending on operating systems, virtualization and hardware refresh to drive-up economic market factors. If the global economic crisis deepens, this anticipated spending spree could be short-lived and its impact shallow.

UCS and VCE vBlock Type 1 Challenge Top VMmark

[caption id="attachment_1306" align="alignright" width="105" caption="VMmark "Tile" Loads "][/caption]

Last November we reported on the Fujitsu RX300 S5 rack server taking the top VMmark spot for 8-core systems. Yesterday (January 11, 2010) Cisco's UCS B200-M1 using VMware ESX 4.0 (build 164009) came within 0.5% of the top spot with a score of 25.06@17 tiles. While falling only slightly short of the mark set by the brute force RX300/DX80 combo, the UCS system did so with a very different solution, unsurprisingly similar to the vBlock Type 1 architecture described by Chad Sakacc in his blog post about the VMware, Cisco and EMC alliance.

Given that VMmark is a single node test harness, the difference between rack server and blade server architectures is a non-issue. However, more than just rack vs. blade is going on in this comparison. The Cisco UCS system is being fed by a pair of 10GE converged network adapters - used both for host network access and Fiber Channel bus access - and a monolithic storage array in the guise of a CLARiiON CX4-240 complete with a complement of 20, 73GB STEC SSD's - just to sweeten the pot.

[caption id="attachment_1385" align="aligncenter" width="450" caption="VMmark Network Configuration for the UCS B200-M1"][/caption]

While it is clear from past VMmark posts that the network speed (beyond 1Gbps) has little to do with the results, it is nice to see the confidence Cisco has in the CNA approach (Cisco UCS M71KR-Q) to go with the "eggs in a basket" solution. Given the storage demands on the CNA, the VMmark result should remove any doubt about the viability (performance) of high-capacity tandems (we'll leave the physical link security concerns for another day.)

However, where the "rubber meets the road" in this contest is storage I/O and this solution - in our opinion is just plain showing off. With just 41 disks to build from, the CX4-240 has been configured to deliver 37 LUNs - nearly one LUN per unit disk. Before any awards are given out for storage of the year, we need to consider that 36 of those LUNs are RAID0 - yielding a testing platform with no real-world analog (hence "showing off".)

[caption id="attachment_1388" align="aligncenter" width="449" caption="CLARiiON CX4-240 Storage Build-out for UCS B200-M1 VMmark"][/caption]

Given the ease at which RAID0 can be replaced by RAID1+0, it may be safe to assume that the same results could have been obtained by using 77 disks instead of 41 - at which point the CX4-240 would still be less than half the size of the top VMmark's 172-disk solution. The reason is clear: SSD's accelerate I/O loads incredibly well in architectures that support them. If anything, this "runner-up" proves that SSD adoption is on the verge of becoming mainstream.

But what does this test show about UCS? Firstly, it shows that Cisco's platform can compete with the best solutions out there on CPU and I/O performance (what's a half a percentage point across 102 virtual machines?) It's not really a surprise given that the UCS platform was designed to do just that - and within a neatly managed framework. Secondly, it shows that the choice of EMC as a partner was an excellent one. As Martin Glassborow commented on his Storagebod's Blog, EMC's involvement in VMware has energized the storage vendor to take bold and innovative steps towards Cloud Computing solutions that it might not have done otherwise (like the RAID0 SSD array). Thirdly and most importantly, it underscores the importance of predictable performance in a virtualization solution. Given the UCS/vBlock approach to systems organization, it can be very difficult not to draw solid parallels between the benchmarks and expectations for net new builds based on the criterion.

NEC Offers "Dunnington" Liposuction, Tops 64-Core VMmark

NEC’s venerable Express5800/A1160 is back at the top VMmark chart, this time establishing the brand-new 64-core category with a score of 48.23@32 tiles - surpassing its 48-core 3rd place posting by over 30%. NEC’s new 16-socket, 64-core, 256GB “Dunnington” X7460 Xeon-based score represents a big jump in performance over its predecessor with a per tile ratio of 1.507 - up 6% from the 48-core ratio of 1.419.

To put this into perspective, the highest VMmark achieved, to date, is the score of 53.73@35 tiles (tile ratio 1.535) from the 48-core HP DL785 G6 in August, 2009. If you are familiar with the "Dunnington" X7460, you know that it's a 6-core, 130W giant with 16MB L2 cache and a 1000's price just south of $3,000 per socket. So that raises the question: how does 6-cores X 16-sockets = 64? Well, it's not pro-rationing from the Obama administration's "IT fairness" czar. NEC chose to disable the 4th and 6th core of each socket to reduce the working cores from 96 to 64.

At $500/core, NEC's gambit may represent an expensive form of "core liposuction" but it was a necessary one to meet VMware's "logical processor per host" limitation of 64. That's right, currently VMware's vSphere places a limit on logical processors based on the following formula:

CPU_Sockets X Cores_Per_Socket X Threads_Per_Core =< 64

According to VMware, the other 32 cores would have been "ignored" by vSphere had they been enabled. Since "ignored" is a nebulous term (aka "undefined"), NEC did the "scientific" thing by disabling 32 cores and calling the system a 64-core server. The win here: a net 6% improvement in performance per tile over the 6-core configuration - ostensibly from the reduced core loading on the 16MB of L3 cache per socket and reduction in memory bus contention.

Moving forward to 2010, what does this mean for vSphere hardware configurations in the wake of 8-core, 16-thread Intel Nehalem-EX and 12-core, 12-thread AMD Magny-Cours processors? With a 4-socket Magny-Cours system limitation, we won't be seeing any VMmarks from the boys in green beyond 48-cores. Likewise, the boys in blue will be trapped by a VMware limitation (albeit, a somewhat arbitrary and artificial one) into a 4-socket, 64-thread (HT) configuration or an 8-socket, 64-core (HT-disabled) configuration for their Nehalem-EX platform - even if using the six-core variant of EX. Looks like VMware will need to lift the 64-LCPU embargo by Q2/2010 just to keep up.

Quick Take: Red Hat and Microsoft Virtual Inter-Op

This week Red Hat and Microsoft announced support of certain of their OSes as guests in their respective hypervisor implementations: Kernel Virtual Machine (KVM) and Hyper-V, respectively. This comes on the heels of Red Hat's Enterprise Server 5.4 announcement last month.

KVM is Red Hat's new hypervisor that leverages the Linux kernel to accelerate support for hardware and capabilities. It was Red Hat and AMD that first demonstrated live migration between AMD and Intel-based hypervisors using KVM late last year - then somewhat of a "Holy Grail" of hypervisor feats. With nearly a year of improvements and integration into their Red Hat Enterprise Server and Fedora "free and open source" offerings, Red Hat is almost ready to strike-out in a commercially viable way.

Microsoft now officially supports the following Red Hat guest operating systems in Hyper-V:

Red Hat Enterprise Linux 5.2, 5.3 and 5.4

Red Hat likewise officially supports the following Microsoft quest operating systems in KVM:

Windows Server 2003, 2008 and 2008 R2

The goal of the announcement and associated agreements between Red Hat and Microsoft was to enable a fully supported virtualization infrastructure for enterprises with Red Hat and Microsoft assets. As such, Microsoft and Red Hat are committed to supporting their respective products whether the hypervisor environment is all Red Hat, all Hyper-V or totally heterogeneous - mixing Red Hat KVM and Microsoft Hyper-V as necessary.

"With this announcement, Red Hat and Microsoft are ensuring their customers can resolve any issues related to Microsoft Windows on Red Hat Enterprise Virtualization, and Red Hat Enterprise Linux operating on Microsoft Hyper-V, regardless of whether the problem is related to the operating system or the virtualization implementation."

- Red Hat press release, October 7, 2009

Many in the industry cite Red Hat's adoption of KVM as a step backwards [from Xen] requiring the re-development of significant amount of support code. However, Red Hat's use of libvirt as a common management API has allowed the change to happen much more rapidly that critics assumptions had allowed. At Red Hat Summit 2009, key Red Hat officials were keen to point out just how tasty their "dog food" is:

Tim Burke, Red Hat's vice president of engineering, said that Red Hat already runs much of its own infrastructure, including mail servers and file servers, on KVM, and is working hard to promote KVM with key original equipment manufacturer partners and vendors.

And Red Hat CTO Brian Stevens pointed out in his Summit keynote that with KVM inside the Linux kernel, Red Hat customers will no longer have to choose which applications to virtualize; virtualization will be everywhere and the tools to manage applications will be the same as those used to manage virtualized guests.

- Xen vs. KVM, by Pam Derringer, SearchDataCenter.com

For system integrators and virtual infrastructure practices, Red Hat's play is creating opportunities for differentiation. With a focus on light-weight, high-performance, I/O-driven virtualization applications and no need to support years-old established processes that are dragging on Xen and VMware, KVM stands to leap-frog the competition in the short term.

SOLORI's Take: This news is good for all Red Hat and Microsoft customers alike. Indeed, it shows that Microsoft realizes that its licenses are being sold into the enterprise whether or not they run on physical hardware. With 20+:1 consolidation ratios now common, that represents a 5:1 license to hardware sale for Microsoft, regardless of the hypervisor. With KVM's demonstrated CPU agnostic migration capabilities, this opens the door to an even more diverse virtualization infrastructure than ever before.

On the Red Hat side, it demonstrates how rapidly Red Hat has matured its offering following the shift to KVM earlier this year. While KVM is new to Red Hat, it is not new to Linux or aggressive early adopters since being added to the Linux kernel as of 2.6.20 back in September of 2007. With support already in active projects like ConVirt (VM life cycle management), OpenNebula (cloud administration tools), Ganeti, and Enomaly's Elastic Computing Platform, the game of catch-up for Red Hat and KVM is very likely to be a short one.

Quick Take: Nehalem/Istanbul Comparison at AnandTech

Johan De Gelas and crew present an interesting comparison of Dunnington, Shanghai, Istanbul and Nehalem in a new post at AnandTech this week. In the test line-up are the "top bin" parts from Intel and AMD in 4-core and 6-core incarnations:

• Intel Nehalem-EP Xeon, X5570 2.93GHz, 4-core, 8-thread


• Intel "Dunnington" Xeon, X7460, 2.66GHz, 6-core, 6-thread


• AMD "Shanghai" Opteron 2389/8389, 2.9GHz, 4-core, 4-thread


• AMD "Istanbul" Opteron 2435/8435, 2.6GHz, 6-core, 6-thread

Most importantly for virtualization systems architects is how the vCPU scheduling affects "measured" performance. The telling piece comes from the difference in comparison results where vCPU scheduling is equalized:

[caption id="attachment_1276" align="aligncenter" width="450" caption="AnandTech's Quad Sockets v. Dual Sockets Comparison. Oct 6, 2009."][/caption]

When comparing the results, De Gelas hits on the I/O factor which chiefly separates VMmark from vAPUS:

The result is that VMmark with its huge number of VMs per server (up to 102 VMs!) places a lot of stress on the I/O systems. The reason for the Intel Xeon X5570's crushing VMmark results cannot be explained by the processor architecture alone. One possible explanation may be that the VMDq (multiple queues and offloading of the virtual switch to the hardware) implementation of the Intel NICs is better than the Broadcom NICs that are typically found in the AMD based servers.

Johan De Gelas, AnandTech, Oct 2009

This is yet another issue that VMware architects struggle with in complex deployments. The latency in "Dunnington" is a huge contributor to its downfall and why the Penryn architecture was a dead-end. Combined with 8 additional threads in the 2P form factor, Nehalem delivers twice the number of hardware execution contexts than Shanghai, resulting in significant efficiencies for Nehalem where small working data sets are involved.

When larger sets are used - as in vAPUS - the Istanbul's additional cores allows it to close the gap to within the clock speed difference of Nehalem (about 12%). In contrast to VMmark which implies a 3:2 advantage to Nehalem, the vAPUS results suggest a closer performance gap in more aggressive virtualization use cases.

SOLORI's Take: We differ with De Gelas on the reduction in vAPUS' data set to accommodate the "cheaper" memory build of the Nehalem system. While this offers some advantages in testing, it also diminishes one of Opteron's greatest strengths: access to cheap and abundant memory. Here we have the testing conundrum: fit the test around the competitors or the competitors around the test. The former approach presents a bias on the "pure performance" aspect of the competitors, while the latter is more typical of use-case testing.

We do not construe this issue as intentional bias on AnandTech's part, however it is another vector to consider in the evaluation of the results. De Gelas delivers a report worth reading in its entirety, and we view this as a primer to the issues that will define the first half of 2010.

Quick Take: HP Blade Tops 8-core VMmark w/OC'd Memory

HP's ProLiant BL490c G6 server blade now tops the VMware VMmark table for 8-core systems - just squeaking past rack servers from Lenovo and Dell with a score of 24.54@17 tiles: a new 8-core record. The half-height blade was equipped with two, quad-core Intel Xeon X5570 (Nehalem-EP, 130W TDP) and 96GB ECC Registered DDR3-1333 (12x 8GB, 2-DIMM/channel) memory.

In our follow-up, we found that HP's on-line configuration tool does not allow for DDR3-1333 memory so we went to the street for a comparison. For starters, we examined the on-line price from HP with DDR3-1066 memory and the added QLogic QMH2462 Fiber Channel adapter ($750) and additional NC360m dual-port Gigabit Ethernet controller ($320) which came to a grand total of $28,280 for the blade (about $277/VM, not including Blade chassis or SAN storage).

Stripping memory from the build-out results in a $7,970 floor to the hardware, sans memory. Going to the street to find 8GB sticks with DDR3-1333 ratings and HP support yielded the Kingston KTH-PL313K3/24G kit (3x 8GB DIMMs) of which we would need three to complete the build-out.  At $4,773 per kit, the completed system comes to $22,289 (about $218/VM, not including chassis or storage) which may do more to demonstrate Kingston's value in the market place rather than HP's penchant for "over-priced" memory.

Now, the interesting disclosure from HP's testing team is this:

[caption id="attachment_1203" align="aligncenter" width="450" caption="Notes from HP's VMmark submission."][/caption]

While this appears to boost memory performance significantly for HP's latest run (compared to the 24.24@17 tiles score back in May, 2009) it does so at the risk of running the Nehalem-EP memory controller out of specification - essentially, driving the controller beyond the rated load. It is hard for us to imagine that this specific configuration would be vendor supported if used in a problematic customer installation.

SOLORI's Take:Those of you following closely may be asking yourselves: "Why did HP choose to over-clock the  memory controller in this run by pushing a 1066MHz, 2DPC limit to 1333MHz?"  It would appear the answer is self-evident: the extra 6% was needed to put them over the Lenovo machine. This issue raises a new question about the VMmark validation process: "Should out of specification configurations be allowed in the general benchmark corpus?" It is our opinion that VMmark should represent off-the-shelf, fully-supported configurations only - not esoteric configuration tweaks and questionable over-clocking practices.

Should there be as "unlimited" category in the VMmark arena? Who knows? How many enterprises knowingly commit their mission critical data and processes to systems running over-clocked processors and over-driven memory controllers? No hands? That's what we thought... Congratulations anyway to HP for clawing their way to the top of the VMmark 8-core heap...

Quick Take: DRAM Price Follow-Up

As anticipated, global DRAM prices have continued their upward trend through September, 2009. We reported on August 4, 2009 about the DDR3 and DDR2 price increases that - coupled with a short-fall in DDR3 production - have caused a temporary shift of the consumer market towards DDR2-based designs.

Last week, the Inquirer also reported that DRAM prices were on the rise and that the trend will result in parity between DDR2 and DDR3 prices. MaximumPC ran the Inquirer's story urging its readers to buy now as the tide rises on both fronts. DRAMeXchange is reporting a significant revenue gain to the major players in the DRAM market as a result of this well orchestrated ballet of supply and demand. The net result for consumers is higher prices across the board as the DDR2/DDR3 production cross-over point is reached.

SOLORI's Take: DDR2 is a fading bargain in the server markets, and DIMM vendors like Kingston are working to maintain a stable source of DDR2 components through the end of 2009. While still Looking at our benchmark tracking components, we project 8GB DIMMs to average $565/DIMM by the end of 2009. In the new year, expect 8GB/DDR2 to hit $600/DIMM by the end of H2/2010 with lower pricing on 8GB/DDR3-1066 - in the $500/DIMM range (if supply can keep up with new system demands created by continued growth in the virtualization market.)

Benchmark Server Memory Pricing
DDR2 Series (1.8V)	Price Jun '09	Price Sep '09	DDR3 Series (1.5V)	Price Jun '09	Price Sep '09
4GB 800MHz DDR2 ECC Reg with Parity CL6 DIMM Dual Rank, x4 (5.4W)	$100.00	$117.00 up 17%	4GB 1333MHz DDR3 ECC Reg w/Parity CL9 DIMM Dual Rank, x4 w/Therm Sen (3.96W)	$138.00	$151.00 up 10%
4GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (5.94W)	$80.00	$103.00 up 29%	4GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (5.09W)	$132.00	$151.00 up 15%
8GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (7.236W)	$396.00	$433.00 up 9%	8GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (6.36W)	$1035.00	$917.00 down 11.5%

SOLORI's 2nd Take: Samsung has been driving the DRAM roller coaster in an effort to dominate the market. With Samsung's 40-nm 2Gb DRAM production ramping by year end, the chip maker's infulence could create a disruptive position in the PC and server markets by driving 8GB/DDR3 prices into the sub-$250/DIMM range by 2H/2010. Meanwhile Hynix, the #2 market leader, chases with 40-nm 1Gb DDR3 giving Samsung the opportunity to repeat its 2008/2009 gambit in 2010 making it increasingly harder for competitors to get a foot-hold in the DDR3 market.

Samsung has their eye on the future with 16GB and 32GB DIMMs already exhibited with 50-nm 2Gb parts claiming a 20% power savings over the current line of memory. With 40-nm 2Gb parts, Samsung is claiming up to 30% additional power savings. To put this into perspective, eight 32GB DIMMs would could about 60% of the power consumed by 32 8GB DIMMs (requiring a 4P+ server). In a virtualization context, this is enough memory to enable 100 virtual machines with 2.5GB of memory each without over subscription. Realistically, we expect to see 16GB DDR3 DIMMs at $1,200/DIMM by 2H/2010 - if everything goes according to plan.

Quick Take: Magny-Cours Spotted, Pushed to 3GHz for wPrime

Andreas Galistel at NordicHardware posted an article showing a system running a pair of engineering samples of the Magny-Cours processor running at 3.0GHz. Undoubtedly these images were culled from a report "leaked" on XtremeSystems forums showing a "DINAR2" motherboard with SR5690 chipset - in single and dual processor installation - running Magny-Cours at the more typical pre-release speed of 1.7GHz.

We know that Magny-Cours is essentially a MCM of Istanbul delivered in the rectangular socket G34 package. One thing illuminating about the two posts is the reported "reduction" in L3 cache from 12MB (6MB x 2 in MCM) to 10MB (2 x 5MB in MCM). Where did the additional cache go? That 's easy: since a 2P Magny-Cours installation is essentially a 4P Istanbul configuration, these processors have the new HT Assist feature enabled - giving 1MB of cache from each chip in the MCM to HT Assist.

"wPrime uses a recursive call of Newton's method for estimating functions, with f(x)=x²-k, where k is the number we're sqrting, until Sgn(f(x)/f'(x)) does not equal that of the previous iteration, starting with an estimation of k/2. It then uses an iterative calling of the estimation method a set amount of times to increase the accuracy of the results. It then confirms that n(k)²=k to ensure the calculation was correct. It repeats this for all numbers from 1 to the requested maximum."

- wPrime site

Another thing intriguing about the XtremeSystems post in particular is the reported wPrime 32M and 1024M completion times. Compared to the hyper-threading-enabled 2P Xeon W5590 (130W TDP) running wPrime 32M at 3.33GHz (3.6GHz turbo)  in 3.950 seconds, the 2P 3.0GHz Magny-Cours completed wPrime 32M in an unofficial 3.539 seconds - about 10% quicker while running a 10% slower clock. From the myopic lens of this result, it would appear AMD's choice of "real cores" versus hyper-threading delivers its punch.

SOLORI's Take: As a "reality check" we can compared the reigning quad-socked, quad-core Opteron 8393 SE result in wPrime 32M and wPrime 1024M at 3.90 and 89.52  seconds, respectively. Adjusted for clock and core count versus its Shanghai cousin, the Magny-Cours engineering samples - at 3.54 and 75.77 seconds, respectively - turned-in times about 10% slower than our calculus predicted. While still "record breaking" for 2P systems, we expected the Magny-Cours/Istanbul cores to out-perform Shanghai clock-per-clock - even at this stage of the game.

Due to the multi-threaded nature of the wPrime benchmark, it is likely that the HT Assist feature - enabled in a 2P Magny-Cours system by default - is the cause of the discrepancy. By reducing the available L3 cache by 1MB per die - 4MB of L3 cache total - HT Assist actually could be creating a slow-down. However, there are several things to remember here:

• These are engineering samples qualified for 1.7GHz operation


• Speed enhancements were performed with tools not yet adapted to Magny-Cours


• The author indicated a lack of control over AMD's Cool 'n Quiet technology which could have made "as tested" core clocks somewhat lower than what CPUz reported (at least during the extended tests)


• It is speculated that AMD will release Magny-Cours at 2.2GHz (top bin) upon release, making the 2.6+ GHz results non-typical


• The BIOS and related dependencies are likely still being "baked"

Looking at the more "typical" engineering sample speed tests posted on the XtremeSystems' forum tracks with the 3.0GHz overclock results at a more "typical" clock speed of 2.6GHz for 2P Magny-Cours: 3.947 seconds and 79.625 seconds for wPrime 32M and 1024M, respectively. Even at that speed, the 24-core system is on par with the 2P Nehalem system clocked nearly a GHz faster. Oddly, Intel reports the W5590  as not supporting "turbo" or hyper-threading although it is clear that Intel's marketing is incorrect based on actual testing.

Assuming Magny-Cours improves slightly on its way to market, we already know how 24-core Istanbul stacks-up against 16-thread Nehalem in VMmark and what that means for Nehalem-EP. This partly explains the marketing shift as Intel tries to position Nehalep-EP as a destined for workstations instead of servers. Whether or not you consider this move a prelude to the ensuing Nehalem-EX v. Magny-Cours combat to come or an attempt to keep Intel's server chip power average down by eliminating the 130W+ parts from the "server" list,  Intel and AMD will each attempt win the war before the first shot is fired. Either way, we see nothing that disrupts the price-performance and power-performance comparison models that dominate the server markets.

[Ed: The 10% difference is likely due to the fact that the author was unable to get "more than one core" clocked at 3.0GHz. Likewise, he was uncertain that all cores were reliably clocking at 2.6GHz for the longer wPrime tests. Again, this engineering sample was designed to run at 1.7GHz and was not likely "hand picked" to run at much higher clocks. He speculated that some form of dynamic core clocking linked to temperature was affecting clock stability - perhaps due to some AMD-P tweaks in Magny-Cours.]

Quick Take: Dell/Nehalem Take #2, 2P VMmark Spot

The new 1st runner-up spot for VMmark in the “8 core” category was taken yesterday by Dell's R710 - just edging-out the previous second spot HP ProLiant BL490 G6 by 0.1% - a virtual dead heat. Equipped with a pair of Xeon X5570 ($1386/ea, bulk list) and 96GB registered DDR3/1066 (12x8GB), the 2U, rack mount R710 weighs-in with a tile ratio of 1.43 over 102 VMs. :

• Dell R710 w/redundant high-output power supply, ($18,209)


• 2 x Intel Xeon X5570 Processors (included)


• 96GB ECC DDR3/1066 (12×8GB) (included)


• 2 x Broadcom NexXtreme II 5709 dual-port GigabitEthernet w/TOE (included)


• 1 x Intel PRO 1000VT quad-port GigabitEthernet (1x PCIe-x4 slot, $529)


• 3 x QLogic QLE2462 FC HBA (1x PCIe slot, $1,219/ea)


• 1 x LSI1078 SAS Controller (on-board)


• 8 x 15K SAS OS drive, RAID10 (included)


• Required ProSupport package ($2,164)


• Total as Configured: $24,559 ($241/VM, not including storage)

Three Dell/EMC CX3-40f arrays were used as the storage backing of the test. The storage system included 8GB cache, 2 enclosures and 15, 15K disks per array delivering 19 LUNs at about 300GB each. Intel's Hyper-Threading and  "Turbo Boost" were enabled for 8-thread, 3.33GHz core clocking as was VT; however embedded SATA and USB were disabled as is common practice.

At about $1,445/tile ($241/VM) the new “second dog” delivers its best at a 20% price premium over Lenovo's "top dog" - although the non-standard OS drive configuration makes-up a half of the difference, with Dell's mandatory support package making-up the remainder. Using a simple RAID1 SAS and eliminating the support package would have droped the cost to $20,421 - a dead heat with Lenovo at $182/VM.

Comparing the Dell R710 the 2P, 12-core benchmark HP DL385 G6 Istanbul system at 15.54@11 tiles:

• HP DL385 G6  ($5,840)


• 2 x AMD 2435 Istanbul Processors (included)


• 64GB ECC DDR2/667 (8×8GB) ($433/DIMM)


• 2 x Broadcom 5709 dual-port GigabitEthernet (on-board)


• 1 x Intel 82571EB dual-port GigabitEthernet (1x PCIe slot, $150/ea)


• 1 x QLogic QLE2462 FC HBA (1x PCIe slot, $1,219/ea)


• 1 x HP SAS Controller (on-board)


• 2 x SAS OS drive (included)


• $10,673/system total (versus $14,696 complete from HP)

Direct pricing shows Istanbul’s numbers at $1,336/tile ($223/VM) which is  a 7.5% savings per-VM over the Dell R710. Going to the street - for memory only - changes the Istanbul picture to $970/tile ($162/VM) representing a 33% savings over the R710.

SOLORI's Take: Istanbul continues to offer a 20-30% CAPEX value proposition against Nehalem in the virtualization use case - even without IOMMU and higher memory bandwidth promised in upcoming Magny-Cours. With the HE parts running around $500 per processor, the OPEX benefits are there for Istanbul too. It is difficult to understand why HP wants to charge $900/DIMM for 8GB PC-5300 sticks when they are available on the street for 50% less - that's a 100% markup. Looking at what HP charges for 8GB DDR3/1066 - $1,700/DIM - they are at least consistent. HP's memory pricing practice makes one thing clear - customers are not buying large memory configurations from their system vendors...

On the contrary, Dell appears to be happy to offer decent prices on 8GB DDR3/1066 with their R710 at approximately $837/DIMM - almost par with street prices.  Looking to see if this parity held up with Dell's AMD offerings, we examined the prices offered with Dell's R805: while - at $680/DIMM - Dell's prices were significantly better than HP's, they still exceeded the market by 50%. Still, we were able to configure a Dell R805 with AMD 2435's for much less than the equivalent HP system:

• Dell R805 w/redundant power ($7,214)


• 2 x AMD 2435 Istanbul Processors (included)


• 64GB ECC DDR2/667 (8×8GB) ($433/ea, street)


• 4 x Broadcom 5708 GigabitEthernet (on-board)


• 1 x Intel PRO 100oPT dual-port GigabitEthernet (1x PCIe slot, included)


• 1 x QLogic QLE2462 FC HBA (1x PCIe slot, included)


• 1 x Dell PERC SAS Controller (on-board)


• 2 x SAS OS drive (included)


• $10,678/system total (versus $12,702 complete from Dell)

This offering from Dell should be able to deliver equivalent performance with HP's DL385 G6 and likewise savings/VM compared to the Nehalem-based R710. Even at the $12,702 price as delivered from Dell, the R805 represents a potential $192/VM price point - about $50/VM (25%) savings over the R710.

Quick Take: HP Plants the Flag with 48-core VMmark Milestones

Following on the heels of last month we predicted that HP could easily claim the VMmark summit with its DL785 G6 using AMD's Istanbul processors:

If AMD’s Istanbul scales to 8-socket at least as efficiently as Dunnington, we should be seeing some 48-core results in the 43.8@30 tile range in the next month or so from HP’s 785 G6 with 8-AMD 8439 SE processors. You might ask: what virtualization applications scale to 48-cores when $/VM is doubled at the same time? We don’t have that answer, and judging by Intel and AMD’s scale-by-hub designs coming in 2010, that market will need to be created at the OEM level.

Well, HP didn't make us wait too long. Today, the PC maker cleared two significant VMmark milestones: crossing the 30 tile barrier in a single system (180 VMs) and exceeding the 40 mark on VMmark score. With a score of 47.77@30 tiles, the HP DL785 G6 - powered by 8 AMD Istanbul 8439 SE processors and 256GB of DDR2/667 memory - set the bar well beyond the competition and does so with better performance than we expected - most likely due to AMD's "HT assist" technology increasing its scalability.

Not available until September 14, 2009, the HP DL785 G6 is a pricey competitor. We estimate - based on today's processor and memory prices - that a system as well appointed as the VMmark-configured version (additional NICs, HBA, etc) will run at least $54,000 or around $300/VM (about $60/VM higher than the 24-core contender and about $35/VM lower than HP's Dunnnigton "equivalent").

SOLORI's Take: While the September timing of the release might imply a G6 with AMD's SR5690 and IOMMU, we're doubtful that the timing is anything but a coincidence: even though such a pairing would enable PCIe 2.0 and highly effective 10Gbps solutions. The modular design of the DL785 series - with its ability to scale from 4P to 8P in the same system - mitigates the economic realities of the dwindling 8P segment, and HP has delivered the pinnacle of performance for this technology.

We are also impressed with HP's performance team and their ability to scale Shanghai to Istanbul with relative efficiency. Moving from DL785 G5 quad-core to DL785 G6 six-core was an almost perfect linear increase in capacity (95% of theoretical increase from 32-core to 48-core) while performance-per-tile increased by 6%. This further demonstrates the "home run" AMD has hit with Istanbul and underscores the excellent value proposition of Socket-F systems over the last several years.

Unfortunately, while they demonstrate a 91% scaling efficiency from 12-core to 24-core, HP and Istanbul have only achieved a 75% incremental scaling efficiency from 24-cores to 48-cores. When looking at tile-per-core scaling using the 8-core, 2P system as a baseline (1:1 tile-to-core ratio), 2P, 4P and 8P Istanbul deliver 91%, 83% and 62.5% efficiencies overall, respectively. However, compared to the %58 and 50% tile-to-core efficiencies of Dunnington 4P and 8P, respectively, Istanbul clearly dominates the 4P and 8P performance and price-performance landscape in 2009.

In today's age of virtualization-driven scale-out, SOLORI's calculus indicates that multi-socket solutions that deliver a tile-to-core ratio of less than 75% will not succeed (economically) in the virtualization use case in 2010, regardless of socket count. That said - even at a 2:3 tile-to-core ratio - the 8P, 48-core Istanbul will likely reign supreme as the VMmark heavy-weight champion of 2009.

SOLORI's 2nd Take: HP and AMD's achievements with this Istanbul system should be recognized before we usher-in the next wave of technology like Magny-Cours and Socket G34. While the DL785 G6 is not a game changer, its footnote in computing history may well be as a preview of what we can expect to see out of Magny-Cours in 2H/2010. If 12-core, 4P system price shrinks with the socket count we could be looking at a $150/VM price-point for a 4P system: now that would be a serious game changer.

Quick Take: 6-core "Gulftown" Nehalem-EP Spotted, Tested

TechReport is reporting on a Taiwanese overclocker who may be testing a pair of Nehalem 6-core processors (2P) slated for release early in 2010. Likewise, AlienBabelTech mentions a Chinese website, HKEPC, that has preliminary testing completed on the desktop (1P) variant of the 6-core. While these could be different 32nm silicon parts, it is more likely - judging from the CPU-Z outputs and provided package pictures - that these are the same sample SKUs tested as 1P and 2P LGA-1366 components.

What does this mean for AMD and the only 6-core shipping today? Since Intel's still projecting Q2/2010 for the server part, AMD has a decent opportunity to grow market share for Istanbul. Intel's biggest rival will be itself - facing a wildly growing number of SKU's in across its i-line from i5, i7, i8 and i9 "families" with multiple speed and feature variants. Clearly, the non-HT version would stand as a direct competitor to Istanbul's native 6-core SKUs. Likewise, Istanbul may be no match for the 6-core Nehalem with HT and "turbo core" feature set.

However, with an 8-core "Beckton" Nehalem variant on the horizon, it might be hard to understand just where the Gulftown fits in Intel's picture. Intel faces a serious production issue, filling fab capacity with 4-core, 6-core and 8-core processors, each with speed, power, socket and HT variants from which to supply high-speed, high-power SKUs and lower-speed, low-power SKUs for 1P, 2P and 4P+ destinations. Doing the simple math with 3 SKU's per part Intel would be offering the market a minimum of 18 base parts according to their current marketing strategy: 9 with HT/turbo, 9 without HT/turbo. For socket LGA-1366, this could easily mean 40+ SKUs with 1xQPI and 2xQPI variants included (up from 23).

SOLORI's take: Intel will have to create some interesting "crippling or pricing tricks" to keep Gulftown from canibalizing the Gainstown market. If they follow their "normal" play book, we prodict the next 10-months will play out like this:

1. Initially there will be no 8-core product for 1P and 2P systems (LGA-1366), allowing for artificially high margins on the 8-core EX chip (LGA-1567), slowing the enevitable canibalization of the 4-core/2P market, and easing production burdens;


2. Intel will silently and abruptly kill Itanium in favor of "hyper-scale" Nehalem-EX variants;


3. Gulftown will remain high-power (90-130W TDP) and be positioned against AMD's G34 systems and Magny-Cours - plotting 12-core against 12-thread;


4. Intel creates a "socket refresh" (LGA-1566?) to enable "inexpensive" 2P-4P platforms from its Gulftown/Beckton line-up in 2H/2010 (ostensibly to maintain parity with G34) without hurting EX;


5. Revised, lower-power variants of Gainstown will be positioned against AMD's C32 target market;


6. Intel will cut SKUs in favor of higher margins, increasing speed and features for "same dollar" cost;


7. Non-HT parts will begin to disappear in 4-core configurations completely;


8. Intel's AES enhancements in Gulftown will allow it to further differentiate itself in storage and security markets;

It would be a mistake for Intel to continue growing SKU count or provide too much overlap between 4-core HT and 6-core non-HT offerings. If purchasing trends soften in 4Q/09 and remain (relatively) flat through 2Q/10, Intel will benefit from a leaner, well differentiated line-up. AMD has already announced a "leaner" plan for G34/C32. If all goes well at the fabs, 1H/2010 will be a good ole fashioned street fight between blue and green.

Quick Take: DDR3 Prices on the Rise

In the current server-class arms race, Intel and AMD have secured separate quarters: Intel's rival QPI architecture coupled to a 3-channel DDR3 memory bus and functional hyper-threading cores (top bin parts) holds the pure performance sector; while AMD's improved Istanbul cores can be delivered 6 at a time and paired with inexpensive DDR2 memory to achieve better price-performance (acquisition). Both solutions deliver about the same economies in power consumption under virtualized loads.

All in all, the Twin² with Xeon L5520 CPUs is the best platform for those seeking an affordable server with an excellent performance/watt ratio at an affordable price. On the other hand, if performance/price is the most important criterion followed by performance/watt, we would probably opt for the six-core Opteron version of the Twin². Supermicro has "a blade killer" avialable with the Twin², especially for those people who like to keep the hardware costs low.

John De Gelas, AnantTech, July 22, 2009

[caption id="" align="alignright" width="335" caption="Global DDR2 and DDR3 Capacity"][/caption]

Meanwhile, the cost differential between DDR3 and DDR2 continues to widen due to increased demand in the notebook sector and reduced supply (capacity). According to DRAMeXchange, the trend will continue into Q4/09 as suppliers are expected to commit up to 30% of capacity to DDR3 by that time.

At the same time, DDR3 prices continue to inch up, by 5% in July, while DDR2 prices have appeared to bottom-out. This trend in DDR3 pricing is consistent across all speed ratings (1066/1333/1600) and, despite artificial downward price pressure from Samsung, has managed to drift upward 20% since May, 2009.

[caption id="" align="alignright" width="311" caption="DDR3 Price Trend, May to August, 2009"][/caption]

Because low-end, lower-priced 2GB DDR3/1066 ($60/stick) memory shows little advantage over 2GB DDR2/800 ($35/stick), the 70% price premium keeps DDR2 in demand. With the added economic pressures of the world economy and cautious growth outlook of manufacturing sector, the cross-over from DDR2 to DDR3 will come at a significant cost: either to the consumer or the supplier.

Until the cross-over, DDR2-based systems will continue to be a favorite in price sensitive applications (i.e. where total system cost plays a significant role in purchasing decisions.) As an example of this economic inequality, let's take the HP DL380 G6 and DL385 G6 as a comparison. Adding 16GB to the DL380 adds about $760 to the price tag (4x4GB DDR3-1066), while adding the same amount of memory to the DL385 adds only $410 (4x4GB DDR2-800). This comparison demonstrates an 85% price premium of DDR3 versus DDR2, a bit higher (percentage wise) than the desktop norm of 70%.

SOLORI's Take: While the cost of memory in desktop systems typically represents a small portion of the overall system cost, the same can not be said for virtualization systems where entry configurations weigh-in at 16GB and often run from 48GB to 72GB in "fully loaded" systems. This, as our calculus has shown, is where the sweet-spot of $/VM is delivered.

In such configurations, the cost of DDR3 memory can tripple the system cost ($6,370 for 2P, L5506 w/12x4GB DDR3-1066R vs. $5,201 for 2P 2427 w/12x4GB DDR2-800). Moving to the higher memory footprint in 2P systems is typically not cost effective because core count cannot keep-up with the memory needs of the virtual machine inventory. However, if it were possible to utilize additional memory in the 2P platform, our benchmark 8GB DDR3-1066 versus DDR2-667 price comparison tells another story. At $900/stick, the cost of 8GB DDR3 is still a 235% premium over 8GB DDR2, making 96GB DDR3 systems (2P Xeon w/HT) nearly $6,200 per server more costly than their DDR2 counterparts (2P Istanbul) based on memory pricing alone.

SOLORI's 2nd Take: We're hoping to see Tyan and Supermicro release SR5690 chipset-based systems - promised in Q3/2009 - to take advantage of this pricing trend and round-out the Istanbul offering before Q1/2010 ushers-in the next wave of multi-core systems. With 10G prices on the decline, we think today's virtualization applications make Istanbul+IOMMU a good price-performance and price-feature fit in the 32-64GB memory footprint space, leaving Nehalem-EP with only the performance niche to its credit. The only question is: where is SR5690?

NEC Adds Top 48-Core, Dell Challenges 24-Core in VMmark Race

NEC's venerable Express5800/A1160 tops the 48-core VMmark category today with a score of 34.05@24 tiles to wrest the title away from IBM who established the category back in June, 2009. NEC's new "Dunnington" X7460 Xeon-based score represents a performance per tile ratio of 1.41 and a tile to core efficiency of 50% using 128GB of ECC DDR2 RAM.

Compared to the leading 24-core "Dunnington" results - held by IBM's x3850 M2 at 20.41@14 tiles - the NEC benchmark sets a scalability factor of 85.7% when moving from 4-socket to 8-socket systems. Both servers from NEC and IBM are scalable systems allowing for multiple chassis to be interconnected to achieve greater CPU-per-system numbers - each scaling in 4-CPU increments - ostensibly for OLTP advantages. The NEC starts at around $70K for 128GB and 48-cores resulting in a $486/VM cost to VMmark.

Also released today, Dell's PowerEdge R905 - with 24 2.8GHz Istanbul cores (8439 SE) and 128GB of ECC DDR2 RAM - secures the number two slot in the 24-category with a posting of 29.51@20 tiles. This represents a tile ratio of 1.475 and tile efficiency of 83.3% for the $29K rack server from Dell at about $240/VM. Compared to its 12-core counterpart, this represents a 91% scalability factor.

If AMD's Istanbul scales to 8-socket at least as efficiently as Dunnington, we should be seeing some 48-core results in the 43.8@30 tile range in the next month or so from HP's 785 G6 with 8-AMD 8439 SE processors. You might ask: what virtualization applications scale to 48-cores when $/VM is doubled at the same time? We don't have that answer, and judging by Intel and AMD's scale-by-hub designs coming in 2010, that market will need to be created at the OEM level.

Based on the performance we're seeing in 8-socket systems relative to 4-socket and the upcoming "massively mult-core" processors in 2010, the law of diminishing returns seems to favor the 4-socket system as the limit for anything but massive OLTP workloads. Even then, we expect to see 48-core in a "4-way" box more efficient than the same number of cores in an 8-way box. The choice in virtualization will continue to be workload biased, with 2P systems offering the best "small footprint" $/VM solution and 4P systems offering the best "large footprint" $/VM solution.

RIP Dunnington: HP's 4P/24-core Istanbul Takes VMmark Summit

HP has simultaneously achieved two near identical VMmark scores with their ProLiant DL585 G6 rack server and ProLiant BL685c G6 blade, claiming the summit from the reigning 24-core champion. Since first establishing the 24-core tier VMmark in September 2009, the Intel "Dunnington" 6-core processor (FSB architecture) has gone unchallenged. Now, with the release of the Opteron 8439SE raising the performance bar and the Opteron 8435 making a clear price-performance case, Dunnington's vacation is over.

Today's Istanbul-based achievements - established in the same memory footprint as the top Dunnington - renders the venerable processor all but obsolete, besting the champ by 4 tiles (24 more virtual machines) with a score-tile ratio of 1.5 for the rack system and 1.46 (same as the Dunnington at 14 tiles) for the blade. Using the HP and IBM on-line configuration tools, we established the retail (on-line) price for each system - down to the Fiber Channel HBA's - and compared them for $/VM value. Here are the results:

	HP DL685 G6	HP BL685c G6	IBM x3850 M2
Processor	4x Opteron 8439SE 2.8GHz	4x Opteron 8435 2.6GHz	4x Xeon X7460 2.67GHz
Memory	128GB (16x8GB PC2-5300 Reg ECC)	128GB (16x8GB PC2-5300 Reg ECC)	128GB (32x4GB PC2-5300 Reg ECC)
LAN Controllers	1x Dual-Port NC371i 1Gbps, 3x Dual-Port NC380T 1Gbps	2x Dual-Port NC532i Flex-10 10Gbs, 1x Dual-Port NC360m 1Gbps	2x Intel PRO 1000PT Dual-Port 1Gbps
HBA	Qlogic QMH2462 Dual-Port FC	Qlogic QMH2462 Dual-Port FC	2x Qlogic QMH2462 Dual-Port FC
OS RAID Controller	HP Smart Array P800	HP Smart Array P400i	HBA
OS Disks	2x 73Gb SAS 10K	2x 73Gb SAS 10K	SAN
On-line Price	$36,862.00	$35,296.00	$34,269.00
On-line w/3rd Party Memory	$28,712.00	$27,356.00	$33,207.00
VMmark Results	29.95@20 tiles	29.19@20 tiles	20.5@14 tiles
VMmark Tile Ratio	1.5	1.46	1.46
Cost/VM Retail	$307.18	$294.13	$407.96
Cost/VM 3rd Party	$239.27	$227.97	$276.73

The results indicate a 21-38% savings per-VM for Istanbul over Dunnington in the 4P/24-core virtualization space. This is bread-and-butter territory for VDI implementations and SQL virtualizations, and Intel's last remaining market place for the Dunnington processor. With the top-bin Istanbul weighing-in with 3% better performance, 18% less power consumption and 30% more capacity against Dunnington at the same price point, Intel's 4P gambit is played-out and Nehalem-EX cannot arrive too soon for Intel.

It is worth asking the question: does the HP ProLiant 4P/24-core offer the best value? The answer depends on the value proposition. From a straight $/VM vantage point, the HP DL385 G6 comparison demonstrated a more economical $182/VM - a difference of $40/VM lower than the BL685c G6 - so the 2P rack system still comes out on top for the absolute bottom-line concious. However, for applications like SQL consolidations, the additional savings in licensing on 4P platforms versus 2P platforms dwarfs this differential.

What is clear: AMD's Istanbul solution will remain unchallenged in the 4P space both in raw performance and in price-performance until Nehalem-EX is delivered. That means if Nehalem-EX does not arrive in Q3/2009, the market will likely wait for Q1/2010 to make any long-term purchasing decisions in anticipation of the new platforms slated to break-in the new year.

Lenovo Claims Top VMmark Spot: 2P, 8C

The new top spot for VMmark in the "8 core" category is now held by Lenovo's R525 G2 rack server with a score of 24.35@17 tiles (tile ratio of 1.43 over 102 VMs). As this server appears to be available in the overseas (China) markets only, we can only estimate the street price of the system used in the benchmark based on the reported build-out at to be around $20,330 per server (street):

• Base Lenovo R525 G2 ($4,900 - 30,000 yuan)


• 2 x Intel Xeon X5570 Processors ($1,500/ea)


• 96GB ECC DDR3/1066 (12x8GB) ($900/DIMM from Kingston)


• 1 x Intel 82575EB dual-port GigabitEthernet (on-board)


• 2 x Intel 82571EB dual-port GigabitEthernet (2x PCIe slot, $150/ea)


• 1 x QLogic QLE2462 FC HBA (1x PCIe slot, $1,300)


• 1 x LSI1078 SAS Controller (on-board)


• 2 x SAS OS drive ($300 est.)

An EMC CX3-40f was used as the storage backing of the test. The storage system included 4GB cache, 4 enclosures and 55 146GB 15K FC disks (10, 15, 15, 15), and 17 LUNs at 100GB each. Interestingly, a Cisco Linksys SR2024 GigabitEthernet switch was used for the network interconnection (about $299/each at NewEgg) which implies that test results are not being influenced on network performance or latency. Given the use of a 2-port FC HBA for storage, iSCSI network performance is not a factor.

At about $1,094/tile ($182/VM) the new "top dog" delivers its best at a 5% price-per-VM premium over Istanbul's only VMmark results (1.41 tile ratio) and an 80% system price premium (assuming memory sourced by third parties).  Since we had to go to the street to configure the Lenovo system, the Istanbul system saves about $1,570 [in mark-up] under similar (non-vendor pricing) circumstances:

• Base HP DL385 G6  ($5,100)


• 2 x AMD 2435 Istanbul Processors (included)


• 64GB ECC DDR2/800 (8x8GB) ($370/DIMM)


• 2 x Broadcom 5709 dual-port GigabitEthernet (on-board)


• 1 x Intel 82571EB dual-port GigabitEthernet (1x PCIe slot, $150/ea)


• 1 x QLogic QLE2462 FC HBA (1x PCIe slot, $1,300)


• 1 x HP SAS Controller (on-board)


• 2 x SAS OS drive (included)


• $9,810/system total (versus $11,378 complete from HP)

Street pricing changes Istanbul's numbers to $892/tile ($149/VM) signifying a 22% per-VM savings and a 52% savings in system price. Given that virtualization systems are generally sold in pairs, this comparison shows that a redundant Istanbul system can be had for less than the cost of a non-redundant Nehalem. For SMB's getting started in virtualization, Istanbul continues to offer a compelling system value proposition over Nehalem.

AMD Istanbul and Intel Nehalem-EP: Street Prices

It's been three weeks after the official launch of AMD's 6-core Istanbul processor and we wanted to take a look at prevailing street prices for the DIY upgrade option.

Istanbul Pricing (Street)

AMD "Istanbul" Opteron™ Processor Family
2400 Series	Price	8400 Series	Price
2.6GHz Six-Core, 6-Thread AMD Opteron 2435 (75W ACP)	$1060.77	2.6GHz Six-Core, 6-Thread AMD Opteron 8435 (75W ACP)	$2,842.14
2.4GHz Six-Core, 6-Thread AMD Opteron 2431 (75W ACP)	$743.74 $699.00	2.4GHz Six-Core, 6-Thread AMD Opteron 8431 (75W ACP)	$2,305.70
2.2GHx Six-Core, 6-Thread AMD Opteron 2427 (75W ACP)	$483.82 $499.99

Nehalem-EP/EX Pricing (Street)

After almost two months on the market, the Nehalem has been on the street long enough to see a 1-3% drop in prices. How does Istanbul stack-up against the Nehalem-EP/Xeon pricing?

Intel "Nehalem" Xeon Processor Family
EP Series	Price	EX Series	Price
2.66GHz Quad-Core, 8-Thread Intel Xeon EP X5550 (95W TDP)	$999.95 $999.99	Quad-Core, 8-Thread Intel Xeon EX	TDB
2.4GHz Quad-Core, 8-Thread Intel Xeon EP E5530 (80W TDP)	$548.66 $549.99	Quad-Core, 8-Thread Intel Xeon EX	TBD
2.26GHz Quad-Core, 8-Thread Intel Xeon EP E5520 (80W TDP)	$400.15 $379.99
2.26GHz Quad-Core, 8-Thread Intel Xeon EP L5520 (60W TDP)	$558.77 $559.99

Compared to the competing Nehalem SKU's, the Istanbul is fetching a premium price. This is likely due to the what AMD perceives to be the broader market that Istanbul is capable of serving (and its relative newness relative to demand, et al). Of course, there are no Xeon Nehalem-EX SKU's in supply to compare against Istanbul in the 4P and 8P segments, but in 2P, it appears Istanbul is running 6% higher at the top bin SKU and 27% higher at the lower bin SKU - with the exception of the 60W TDP part, upon which Intel demands a 13% premium over the 2.2GHz Istanbul part.

This last SKU is the "green datacenter" battleground part. Since the higher priced 2.6GHz Istanbul rates a 15W (ACP) premium over the L5520, it will be interesting to see if system integrators will compare it to the low-power Xeon in power-performance implementations. Comparing SPECpower_ssj2008 between similarly configured Xeon L5520 and X5570, the performance-per-watt is within 2% for relatively anemic, dual-channel 8GB memory configurations.

In a virtualization system, this memory configuration would jump from an unusable 8GB to at least 48GB, increasing average power consumption by another 45-55W and dropping the performance-per-watt ratio by about 25%. Looking at the relative performance-per-watt of the Nehalem-EP as compared to the Istanbul in TechReport's findings earlier this month, one could extrapolate that the virtualization performance-per-watt for Istanbul is very competitive - even with the lower-power Xeon - in large memory configurations. We'll have to wait for similar SPECpower_ssj2008 in 4P configurations to know for sure.

System Memory Pricing (Street)

System memory represents 15-20% of system pricing - more in very large memory foot prints. We've indicated that Istanbul's time-to-market strategy shows a clear advantage (CAPEX) in memory pricing alone - more than compensating for the slight premium in CPU pricing.

System Memory Pricing
DDR2 Series (1.8V)	Price	DDR3 Series (1.5V)	Price
4GB 800MHz DDR2 ECC Reg with Parity CL6 DIMM Dual Rank, x4 (5.4W)	$100.00	4GB 1333MHz DDR3 ECC Reg w/Parity CL9 DIMM Dual Rank, x4 w/Therm Sen (3.96W)	$138.00
4GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (5.94W)	$80.00	4GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (5.09W)	$132.00
8GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (7.236W)	$396.00	8GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (6.36W)	$1035.00

These parts show a 28%, 40% and 62% premium price for DDR3 components versus DDR2 which indicates Istanbul's savings window is still wide-open. Since DDR3 prices are not expected to fall until Q3 at the earliest, this cost differential is expected to influence "private cloud" virtualization systems more strongly. However, with the 0.3V lower voltage requirement on the DDR3 modules, Nehalem-EP actually has a slight adavantage from a operational power perspective in dual-channel configurations. When using tripple-channel for the same memory footprint, Nehalem-EP's memory consumes about 58% more power (4x8GB vs. 9x4GB).