Wednesday, December 30, 2009

Looking at memory prices one last time before the year is out and prices of our "benchmark" Kingston DDR3 server DIMMs are on the decline. While the quad rank 8G DDR3/1066 DIMMs are below the $565 target price (at $514) we predicted back in August, the dual rank equivalent (on our benchmark list) are still hovering around $670 each. Likewise, while retail price on the 8G DDR2/667 parts continue to rise, inventory and promotional pricing has managed to keep them flat at $433 each, giving large foot print DDR2 systems a $2,000 price advantage (based on 64GB systems).






























Benchmark Server (Spot) Memory Pricing - Dual Rank DDR2 Only
DDR2 Reg. ECC Series (1.8V)Price Jun '09Price Sep '09Price
Dec '09







KVR800D2D4P6/4G
4GB 800MHz DDR2 ECC Reg with Parity CL6 DIMM Dual Rank, x4
(5.400W operating)

$100.00$117.00
up 17%
$140.70
up 23%


(Promo price, retail $162)








KVR667D2D4P5/4G
4GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (5.940W operating)

$80.00$103.00
up 29%
$97.99
down 5%

(retail $160)







KVR667D2D4P5/8G
8GB 667MHz DDR2 ECC Reg with Parity CL5 DIMM Dual Rank, x4 (7.236W operating)

$396.00$433.00$433.00

(Promo price, retail $515)































Benchmark Server (Spot) Memory Pricing - Dual Rank DDR3 Only
DDR3 Reg. ECC Series (1.5V)Price Jun '09Price Sep '09Price
Dec '09







KVR1333D3D4R9S/4G
4GB 1333MHz DDR3 ECC Reg w/Parity CL9 DIMM Dual Rank, x4 w/Therm Sen (3.960W operating)

$138.00$151.00
up 10%

$135.99

down 10%







KVR1066D3D4R7S/4G
4GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (5.09W 5.085W operating)

$132.00$151.00
up 15%
$137.59
down 9%(retail $162)








KVR1066D3D4R7S/8G
8GB 1066MHz DDR3 ECC Reg w/Parity CL7 DIMM Dual Rank, x4 w/Therm Sen (6.36W 4.110W operating)

$1035.00$917.00
down 11.5%
$667.00
down 28%

(avail. 1/10)

As the year ends, OEMs are expected to "pull up inventory," according to DRAMeXchange, in advance of a predicted market short fall somewhere in Q2/2010. Demand for greater memory capacities are being driven by Windows 7 and 64-bit processors with 4GB as the well established minimum system foot print ending 2009. With Server 2008 systems demanding 6GB+ and increased shift towards large memory foot print virtualization servers and blades, the market price for DDR3 - just turning the corner in Q1/2010 versus DDR2 - will likely flatten based on growing demand.

SOLORI's Take: With Samsung and Hynix doubling CAPEX spending in 2010, we'd be surprised to see anything more than a 30% drop in retail 4GB and 8GB server memory by Q3/2010 given the anticipated demand. That puts 8G DDR3/10666 at $470/stick versus $330 for 2x 4GB and on track with August 2009 estimates. The increase in compute, I/O and memory densities in 2010 will be market changing and memory demand will play a small (but significant) role in that development.

In the battle to "feed" the virtualization servers of 2H/2010, the 4-channel "behemoth" Magny-Cours system could have a serious memory/price advantage with 8 (2-DPC) or 12 (3-DPC) configurations of 64GB (2.6GB/thread) and 96GB (3.9GB/thread) DDR3/1066 using only 4GB sticks (assumes 2P configuration). Similar GB/thread loads on Nehalem-EP6 "Gulftown" (6-core/12-thread) could be had with 72GB DDR3/800 (18x 4GB, 3-DPC) or 96GB DDR3/1066 (12x 8GB, 2-DPC), providing the solution architect with a choice between either a performance (memory bandwidth) or price (about $2,900 more) crunch. This means Magny-Cours could show a $2-3K price advantage (per system) versus Nehalem-EP6 in $/VM optimized VDI implementations.

Where the rubber starts to meet the road, from a virtualization context, is with (unannounced) Nehalem-EP8 (8-core/16-thread) which would need 96GB (12x 8GB, 2-DPC) to maintain 2.6GB/thread capacity with Magny-Cours. This creates a memory-based price differential - in Magny-Cours' favor - of about $3K per system/blade in the 2P space. At the high-end (3.9GB/thread), the EP8 system would need a full 144GB (running DDR3/800 timing) to maintain GB/thread parity with 2P Magny-Cours - this creates a $5,700 system price differential and possibly a good reason why we'll not actually see an 8-core/16-thread variant of Nehalem-EP in 2010.

Assuming that EP8 has 30% greater thread capacity than Magny-Cours (32-threads versus 24-threads, 2P system), a $5,700 difference in system price would require a 2P Magny-Cours system to cost about $19,000 just to make it an even value proposition. We'd be shocked to see a MC processor priced above $2,600/socket, making the target system price in the $8-9K range (24-core, 2P, 96GB DDR3/1066). That said, with VDI growth on the move, a 4GB/thread baseline is not unrealistic (4 VM/thread, 1GB per virtual desktop) given current best practices. If our numbers are conservative, that's a $100 equipment cost per virtual desktop - about 20% less than today's 2P equivalents in the VDI space. In retrospect, this realization makes VMware's decision to license VDI per-concurrent-user and NOT per socket a very forward-thinking one!

Of course, we're talking about rack servers and double-size and non-standard blades here: after all, where can we put 24 DIMM slots (2P, 3-DPC, 4-channel memory) on a SFF blade? Vendors will have a hard enough time with 8-DIMM per processor (2P, 2-DPC, 4-channel memory) configurations today. Plus, all that dense compute and I/O will need to get out of the box somehow (10GE, IB, etc.) It's easy to see that HPC and virtualization platforms demands are converging, and we think that's good for both markets.

SOLORI's 2nd Take: Why does 8GB of DRAM require less than 4GB at the same speed and voltage??? The 4GB stick is based on 36x 256M x 4-bit DDR3-1066 FBGA’s (60nm) and the 8GB stick is based on 36x 512M x 4-bit DDR3-1066 FBGA’s (likely 50nm). According to SAMSUNG, the smaller feature size offers nearly 40% improvement in power consumption (per FBGA). Since the sticks use the same number of FBGA components (1Gb vs 2Gb), the 20% power savings seems reasonable.

The prospect of lower power at higher memory densities will drive additional market share to modules based on 2Gb DRAM modules. The gulf between DDR2 will continue to expand as tooling shifts to majority-DDR3 production and the technology. While minority leader Hynix announced a 50nm 2Gb DDR2 part earlier this year (2009), the chip giant Samsung continues to use 60-nm for its 2Gb DDR2. Recently, Hynix announced a successful validation of its 40-nm class 2Gb DDR3 module operating at 1333MHz and saving up to 40% power from the 50nm design. Similarly, Samsung's leading the DRAM arms race with 30nm, 4Gb DDR3 production which will show-up in 1.35V, 16GB UDIMM and RDIMM in 2010 offering additional power saving benefits over 40-50nm designs. Meanwhile, Samsung has all but abandoned advances on DDR2 feature sizes.

The writing is on the wall for DDR2 systems: unit costs are rising, demand is shrinking, research is stagnant and a new wave of DDR3-based hardware is just over the horizon (1H/2010). While these things will show the door to DDR2-based systems (which enjoyed a brief resurgence in 2009 due to DDR3 supply problems and marginal power differences) as demand and DDR3 advantages heat-up in later 2010, it's kudos to AMD for calling the adoption curve, spot on!

Saturday, December 5, 2009

vSphere, Hardware Version 7 and Hot Plug

VMware's vSphere added hot plug features in hardware version 7 (first introduced in VMware Workstation 6.5) that were not available in the earlier version 4 virtual hardware. Virtual hardware version 7 adds the following new features to VMware virtual machines:

  • LSI SAS virtual device - provides support for Windows Server 2008 fail-over cluster configurations

  • Paravirtual SCSI devices - recently updated to allow booting, can allow higher-performance (greater throughput and lower CPU utilization) than the standard virtual SCSI adapter - especially in SAN environments where I/O-intensive applications are used. Currently supported in Windows Server 2003/2008 and Red Hat Linux 5 - although any version of Linux could be modified to support PVSCSI.

  • IDE virtual device - useful for older OSes that don't support SCSI drivers

  • VMXNET 3 - next generation Vmxnet device with enhanced performance and enhanced networking features.

  • Hot plug virtual devices, memory and CPU - supports hot add/remove of virtual devices, memory and CPU for supported OSes.


While the "upgrade" process from version 4 to version 7 is well-known, some of the side effects are not well publicised. The most obvious change after the migration from version 4 to version 7 is the affect hot plug has on the PCI bus adapters - some are now hot plug by default, including the network adapters!

[caption id="attachment_1342" align="aligncenter" width="417" caption="Safe to remove network adapters. Really?"]Safe to remove network adapters. Really?[/caption]

Note that the above example demonstrates also that the updated hardware re-enumerates the network adapters (see #3 and #4) because they have moved to a new PCI bus - one that supports hot plug. Removing the "missing" devices requires a trip to device manager (set devmgr_show_nonpresent_devices=1 in your shell environment first.) This hot plug PCI bus also allows for an administrator to mistakenly remove the device from service - potentially disconnecting tier 1 services from operations (totally by accident, of course.

[caption id="attachment_1340" align="aligncenter" width="450" caption="Devices that can be added while the VM runs with hardware version 4"]Devices that can be added while the VM runs with hardware version 4[/caption]

In virtual hardware version 4, only SCSI devices and hard disks were allowed to be added to a running virtual machine. Now with hardware version 7,

[caption id="attachment_1341" align="aligncenter" width="450" caption="Devices that can be added while the VM runs with hardware version 7"]Devices that can be added while the VM runs with hardware version 7[/caption]

additional devices (USB and Ethernet) are available for hot add. You could change memory and CPU on the fly too, if the OS supports that feature and they are enabled in the virtual machine properties prior to running the VM:

[caption id="attachment_1339" align="aligncenter" width="450" caption="CPU and Memory Hot Plug Properties"]CPU and Memory Hot Plug Properties[/caption]

However, the hot plug NIC issue isn't discussed in the documentation, but Carlo Costanzo at VMwareInfo.com passes on Chris Hahn's great tip to disable hot plug behaviour in his blog post complete with visual aids. The key is to add a new "Advanced Configuration Parameter" to the virtual machine configuration: this new parameter is called "devices.hotplug" and its value should be set to "false." However, adding this parameter requires the virtual machine to be turned-off, so it is currently an off-line fix.