Installing FreeNAS to USB Flash: Easy as 1,2,3

I don't want to get too deep into a re-hash of how to install FreeNAS onto USB flash (or "thumb") drives - there is a wealth of community information in that regard. However, any time I come across the same simple question so many times in one week I have to investigate it more thoroughly.  This week, that question has been "have you tried FreeNAS?"

Anyone familiar with the fine m0n0wall project (and off-shoot pfSense) will instantly recognize the FreeBSD appliance approach taken for FreeNAS. The look-and-feel is very M0n0wall-ish as well. In short, this is a no-nonsense and easy-to-install appliance-oriented distribution that covers the basics of network attached storage: CIFS, NFS and iSCSI. Given that M0n0wall and pfSense both virtualize very well, I have no doubt the VMware appliance version performs likewise.

1, 2, 3... NAS

That said, let's quickly run-down a 1, 2, 3 approach for booting FreeNAS to hardware from USB drive... This is a run-from-ram-disk appliance, so the size of the USB storage device is minimal: about 50-60MB. Since I am still testing the Tyan Transport GT28 system, I will catalog my steps for that platform:

Tyan S2935-SI to be released???

While following-up on the availability issue with GT28 systems, I stumbled-over a question fielded by Tyan's technical support group about hypertransport v3 (HT3) interconnects and the S2935. The GT28 is still being front-page-listed on Tyan's home site making it one of their top three barebones systems.  The S2935 motherboard in the GT28 was also noted in a recent press release as one of the AMD 45nm-ready systems. Tyan support group indicated that the S2935 has been updated to "S2935-SI" (release TBD) to include support for HT3 at some future date:

The S2935 was designed before HT3 (HyperTransport interconnect) technology was announced. The S2935 does not have HT3 link capability. If you want to have HT3 link capability then you can use all the same components on the S2935-SI product instead. This product was re-designed to support HT3 link technologies - Revised 10/22/2008

This is good news for those of you shopping for Transport GT28 systems today and finding short supply: it looks like a new release is on the horizon. The last GT28 system we got took two weeks to fulfill and since then NewEgg has removed the part number from active inventory due to the shortage.

Installing VMware ESXi on the Tyan Transport GT28

Once the GT28 nodes are BIOS-updated to the AGESA v3.3.0.0+ release, a few adjustments are needed to to support my boot-from-flash deployment model. If you are not familiar with boot-from-USB-flash, there are many helpful blogs issued on the subject like this one from vm-help.com. Suffice to say, boot-from-USB-flash for ESXi in a relatively simple process to setup:

1. Make sure your BIOS supports boot-from-USB-flash;


2. Download the latest release of ESX 3i from VMware;


3. Mount the ISO image of the 3i installer;


4. Find the "VMvisor-big" image as a ".dd.bz2" file in the mounted image;


5. Un-bzip the VMvisor-big image to a temporary directory;


6. Plug-in your "donor" USB flash device (I'm using the Sandisk Cruzer 4GB);


7. Find the device handle of the mounted USB device and unmount it (i.e. "umount /dev/sdm";


8. Use dd to copy the VMvisor image to the flash device (i.e. "dd if=/tmp/VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd of=/dev/sdm");


9. Eject the USB device and label it as ESXi;


10. Insert the USB flash device into a USB 2.0 port on your equipment and boot;

Preparing the BIOS

To prepare my GT28 for ESX 3i and boot-from-USB-flash, I insert the USB "thumb drive" into one of the rear ports and turn-on the GT28. Hitting the "delete" key on boot gets me to the BIOS setup.  I will start with the BIOS "Optimal Defaults", and make modifications from there; these adjustments are (follow links for screen shots):

[caption id="attachment_168" align="aligncenter" width="300" caption="S2935 BIOS screen on boot"][/caption]

1. Reset BIOS to "Optimal Defaults";


2. Adjust Northbridge IOMMU window from 128MB to 256MB;


3. Disable AMD PowerNow in BIOS;


4. Adjust PCI Latency Timer from 64 to 128 (optional);


5. Disable nVidia MCP55 SATA controller (ESXi has no driver ESXi has a driver, however may be issues with nVRAID);


6. Adjust USB Mass Storage, setting the USB flash drive to Hard Disk;


7. Disable the CD/DVD boot devices to avoid boot conflicts;


8. Select the USB flash drive as the only boot device;


9. Finally, save the BIOS changes and reboot;


10. Now, the system should boot into ESXi for initial configuration;

As you can see, boot-from-USB-flash is "wicked simple" to implement (at least on this platform) and open-up all kinds of testing scenarios. In this case, the ESXi image is now running from USB flash, and only the basic configuration tasks remain. However, it is a good idea to know which Ethernet ports are which on the rear panel of the GT28.

[caption id="attachment_91" align="aligncenter" width="407" caption="S2935 I/O Ports, Rear"][/caption]

If the PCI bus scan order is configured for "Ascent" the LAN ports will be configured as indicated in the image shown. If you modify the bus scan for "Descent" (i.e. to accommodate a RAID controller) then E2/E3 becomes E0/E1 and E0/E1 becomes E2/E3 due to the new initialization sequence. You may want to, therefore, be cautious when making such a change since ESXi will re-enumerate the interfaces (although any used interface will be pinned to the MAC address.)

Initial Configuration of ESXi

Once your network connections are plugged-in, you should have already mapped-out some IP assignments and VLAN and/or trunking arrangements. While these steps are not strictly necessary in testing, they are a good practice maintain even in testing. To make the initial configurations to ESXi, from the console do the following:

[caption id="attachment_185" align="aligncenter" width="300" caption="S2935 ESXi demo, Initial Configuration"][/caption]

1. Hit "F2" to enter the configuration screen;


2. Set the "root" password for the ESXi server;


3. Skip "Lockdown mode" for now;


4. Configure the management network of the ESXi server;
   
   
   
   1. Select the network adapter(s) to be used for management;
   
   
   2. If not using DHCP:
      
      
      
      1. Fix the management IP address
      
      
      2. Fix the management IP Subnet mask
      
      
      3. Fix the management IP Default gateway
      
      
      4. Fix the management DNS configuration;
      
      
      5. Update the DNS suffix(es) for your local network;
      
      
      
      
   
   
   3. Hit "Enter" to save ("Escape" exits without change);
   
   
   
   


5. Test the management network and restart if necessary;


6. Exit the configuration menu and hit "F12" to restart;

[caption id="attachment_187" align="aligncenter" width="300" caption="S2935 ESXi demo, Restarting ESXi after Configuration"][/caption]

Initial Management with VI Client

Once the ESXi server is up and online, you will need to grab the VMware Infrastructure Client from the web service provided by the ESXi server (http://<ESXi_IP_ADDRESS>/) and install it on your Windows client. If you don't run windows (like me) you should have a version running in a VM for just such an occasion. I find VirtuaBox to be a better (free) choice for workstation-on-workstation applications and VMware Server a good choice if the client is to be minimal and accessible from multiple hosts.

[caption id="attachment_191" align="aligncenter" width="300" caption="S2935 ESXi demo, VMware Infrastructure Client Login"][/caption]

Once the VI Client is installed, run the application and enter the ESXi server's hostname/IP-address, root username and root password where requested. A VI Client window will eventually open and allow you to complete the setup of the ESXi server as needed.

That's really all there is to it: we have a reliable, running ESXi platform in mere minutes with minimal effort.

Notes:

Updated January 15, 2009. Corrected statement that ESX 3i update 3 is not MCP55 aware - support has been added in ESX 3i update 2 and newer. In my test configuration (with the SATA controller enabled) ESX 3i update 3 does properly identify and configure the MCP55 SATA Controller as  an abstracted SCSI controller.

# vmkvsitools lspci
00:00.00 Memory controller: nVidia Corporation
00:01.00 Bridge: nVidia Corporation
00:01.01 Serial bus controller: nVidia Corporation
00:02.00 Serial bus controller: nVidia Corporation
00:02.01 Serial bus controller: nVidia Corporation
00:05.00 Mass storage controller: nVidia Corporation MCP55 SATA Controller [vmhba0]
00:06.00 Bridge: nVidia Corporation
00:10.00 Bridge: nVidia Corporation
00:11.00 Bridge: nVidia Corporation
00:12.00 Bridge: nVidia Corporation
00:13.00 Bridge: nVidia Corporation
00:14.00 Bridge: nVidia Corporation
00:15.00 Bridge: nVidia Corporation
00:24.00 Bridge: Advanced Micro Devices [AMD]
00:24.01 Bridge: Advanced Micro Devices [AMD]
00:24.02 Bridge: Advanced Micro Devices [AMD]
00:24.03 Bridge: Advanced Micro Devices [AMD]
00:24.04 Bridge: Advanced Micro Devices [AMD]
01:05.00 Display controller:
04:00.00 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic0]
04:00.01 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic1]
05:00.00 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic2]
05:00.01 Network controller: Intel Corporation 82571EB Gigabit Ethernet Controller [vmnic3]

This enables the use of the MCP55 SATA controller for flash drives at least. I will do further tests on this platform to determine the stability of the NVRAID component and its suitability for local storage (i.e. embedded VM, like virtual SAN/NAS) needs.

Tyan Transport GT28 Overview, Part 2

As I indicated in the earlier posts, the GT28 uses narrow motherboards within a single 1U chassis to create a compact, dual-node HPC platform. Small as it is, the motherboard still packs two socket-F processors, sixteen 240-pin DDR2 DIMM slots, four GigabitEthernet ports, space for an embedded Mellanox Infiniband processor, on board video, a single low-profile PCIe riser slot (x8 slot/signal) and an SMDC slot.

[caption id="attachment_90" align="alignnone" width="389" caption="Tyan S2935 Motherboard"][/caption]

The GT28 is not a quiet machine once powered-up, and with eight 15,000 RPM fans (3-per node plus 2 power supply fans) the system only gets louder under heavy load. Fortunately, the 45nm Opteron processor is easier on the thermal envelope than its predecessors so the fans stay around 9,000 RPM most of the time. That said, you do not want one of these systems in anything but a full rack enclosure as the tell-tale whine of the system fans is not conducive to office work. This is no appliance chassis and it was not designed to be one, but compared to an 8U AIC storage chassis, this thing is quiet.

The GT28 does provide more than your typical standard compact server motherboard in terms of I/O options - especially network. Although we were not testing the Infiniband variant (about $600 additional) this $1,600 barebones systems comes well equipped for a variety of tasks. As I indicated before, the test system is to become part of a four node storage and hypervisor system based on VMware ESXi and Nexenta NAS (using Solaris' ZFS storage) . One node will run ESXi with Nexenta running in a virtual machine and three nodes will run ESXi using both NFS and iSCSI as the virtual shared storage medium (provided by Nexenta).

I do want to drift briefly into a dicsussion on cost. Although rock-bottom prices are not a focus, per-node costs are - especially where relative to non-commodity computing models.

Tyan Transport GT28 Overview, Part 1

First, let me say that I have been using Tyan Barebones Server products for many years. I have found them both robust in features and reliable. Furthermore, I have focused on AMD Opteron Eco-System products since the products became available due to the I/O benefits of hypertransport and the overall better price/performance of Opteron vs. Xeon offerings.

While I have considerable experience with Xeon systems and have participated in many side-by-side comparisons, I am convinced - as a result of such testing - the I/O systems in Opteron-based platforms are far superior than comparable Xeon FSB systems (front-side bus dependent systems). For low TCO systems, the ability to load I/O elegantly is not just an advantage: it's a must. This loading factors considerably into TCO where per-node utilization and efficiency are large factors.

That said, the GT28 has a Xeon-based counterpart - the Tank GT24 - to satisfy Xeon-based Eco-Systems. The Tank GT24 does not support quad Gigabit Ethernet nor does it support the built-in Mellanox Infiniband interface. The lack of support for these advanced I/O capabilities is further testament to the weak I/O support of the FSB paradigm.

Testing the Tyan Transport GT28 (B2935G28V4H)

Over the next few posts I'll be investigating the Tyan GT28, 1U, 2-socket, dual-node (4-socket total) barebones system. This cost effective system comes some impressive features built-in:

• 1U, Dual-node Chassis (shared 1000W supply)


• Dual AMD Opteron 2000-series processor (per node)


• Shanghai processor support, (AGESA v3.3.0.0 required, BIOS v1.03+)


• 16 DRAM slots (per node, DDR2/REG/ECC)


• 2 x Dual Gigabit Ethernet (per node, Intel 82571EB)


• 20Gbps Infiniband 4x port (per node, B2935G28VHI only, Mellanox MT25204)


• 2 x Hot-Swap SATA/SAS bays (per node)


• 1 x Low-profile PCI Express x 8 slot (per node)


• XGI Volari Z9S Graphics Controller


• nVIDIA nForce Professional 3600 (NFP 3600)


• IPMI/SMDC slot (per node, Tyan M3295-2/M3296)

I am testing mine with the M3296 SMDC card with IP/KVM support (Raritan KIRA100, firmware version 1.00, build 5772, GT28 r01). WARNING: The units I received did not come with AGESA v3.3.0.0 updated BIOS and would not boot with a Shanghai processor - requiring a BIOS update with an older 2000-series chip.

After BIOS update on both the SMDC and motherboard, system boots without issue in my test configuration (per node):

• 4 x Kingston 4GB 240-Pin DDR2 SDRAM DDR2 667 (PC2 5300) ECC Registered Server Memory Model KVR667D2D4P5/4G


• 1 x AMD Opteron 2376 Shanghai 2.3GHz 45nm Socket F 75W Quad-Core Server Processor


• 1 x LSI LSI00110 8-lane, 2.5Gb/s PCI Express SATA / SAS SAS3442E-R 3Gb/s 8-port


• 1 x Tyan M3296 IP/KVM SMDC Card (IPMI 2.0)

I'll have some screen shots of BIOS settings and configurations (courtesy of the M3296's) soon. Since the Gigabit Ethernet controllers support both "jumbo frame" and TCP off-load, we'll try to squeeze-in a few performance comparisons of these features as well.

SME Stack V0.1, Part 3 - Storage Solutions

If storage is the key, shared storage the key that opens all locks. In the early days of file servers, shared storage meant a common file store presented to users over the network infrastructure. This storage was commonly found in a DAS array - usually RAID1 or RAID5 - and managed by a general purpose server operating system (like Windows or Netware). Eventually, such storage paradigms adopted clustering technologies for high-availability, but the underlying principles remained much the same: an extrapolation of a general purpose implementation.

Today, shared storage means something completely different. In fact, the need for "file servers" of old has not disappeared but the dependency on DAS for the place where stored data is ultimately placed has moved to the network. Network attached storage - in the form of filers and network block devices - are replacing DAS as companies retire legacy systems and expand their data storage and business continuity horizons. Today, commercial and open source software options abound that provide stability, performance, scalability, redundancy and feature sets that can provide increased functionality and accelerated ROI to their adopters.