Showing posts with label Open Source Storage. Show all posts
Showing posts with label Open Source Storage. Show all posts

Tuesday, November 16, 2010

Short-Take: New Oracle/Sun ZFS Goodies

I wanted to pass-on some information posted by Joerg Moellenkamp at c0t0d0s0.org - some good news for Sun/ZFS users out there about Solaris Express 2010.11 availability, links to details on ZFS encryption features in Solaris 11 Express and clarification on "production use" guidelines. Here's the pull quotes from his posting:
"Darren (Moffat) wrote three really interesting articles about ZFS encryption: The first one is Introducing ZFS Crypto in Oracle Solaris 11 Express. This blog entry gives you a first overview how to use encryption for ZFS datasets. The second one..."

-  Darren Moffat about ZFS encryption, c0t0d0s0.org, 11-16-2010


"There is a long section in the FAQ about licensing and production use: The OTN license just covers development, demo and testing use (Question 14) . However you can use Solaris 11 Express on your production system as well..."

- Solaris 11 Express for production use, c0t0d0s0.org, 11-16-2010


"A lot of changes found their way into the newest release of Solaris, the first release of Oracle Solaris Express 2010.11. The changes are summarized in a lengthy document, however..."

- What's new for the Administrator in Oracle Solaris  Express 2010.11, c0t0d0s0.org, 11-15-2010



Follow the links to Joerg's blog for more details and links back to the source articles. Cheers!

In-the-Lab: NexentaStor vs. Grub

In this In-the-Lab segment we're going to look at how to recover from a failed ZFS version update in case you've become ambitious with your NexentaStor installation after the last Short-Take on ZFS/ZPOOL versions. If you used the "root shell" to make those changes, chances are your grub is failing after reboot. If so, this blog can help, but before you read on, observe this necessary disclaimer:
NexentaStor is an appliance operating system, not a general purpose one. The accepted way to manage the system volume is through the NMC shell and NMV web interface. Using a "root shell" to configure the file system(s) is unsupported and may void your support agreement(s) and/or license(s).

That said, let's assume that you updated the syspool filesystem and zpool to the latest versions using the "root shell" instead of the NMC (i.e. following a system update where zfs and zpool warnings declare that your pool and filesystems are too old, etc.) In such a case, the resulting syspool will not be bootable until you update grub (this happens automagically when you use the NMC commands.) When this happens, you're greeted with the following boot prompt:
grub>

Grub is now telling you that it has no idea how to boot your NexentaStor OS. Chances are there are two things that will need to happen before your system boots again:

  1. Your boot archive will need updating, pointing to the latest checkpoint;

  2. Your master boot record (MBR) will need to have grub installed again.


We'll update both in the same recovery session to save time (this assumes you know or have a rough idea about your intended boot checkpoint - it is usually the highest numbered rootfs-nmu-NNN checkpoint, where NNN is a three digit number.) The first step is to load the recovery console. This could have been done from the "Safe Mode" boot menu option if grub was still active. However, since grub is blown-away, we'll boot from the latest NexentaStor CD and select the recovery option from the menu.

Import the syspool


Then, we login as "root" (empty password.) From this "root shell" we can import the existing (disks connected to active controllers) syspool with the following command:
# zpool import -f syspool

Note the use of the "-f" card to force the import of the pool. Chances are, the pool will not have been "destroyed" or "exported" so zpool will "think" the pool belongs to another system (your boot system, not the rescue system). As a precaution, zpool assumes that the pool is still "in use" by the "other system" and the import is rejected to avoid "importing an imported pool" which would be completely catastrophic.

With the syspool imported, we need to mount the correct (latest) checkpointed filesystem as our boot reference for grub, destroy the local zfs.cache file (in case the pool disks have been moved, but still all there), update the boot archive to correspond to the mounted checkpoint and install grub to the disk(s) in the pool (i.e. each mirror member).

List the Checkpoints


# zfs list syspool

From the resulting list, we'll pick our highest-numbered checkpoint; for the sake of this article let's say it's "rootfs-nmu-013" and mount it.

Mount the Checkpoint


# mkdir /tmp/syspool
# mount -F zfs syspool/rootfs-nmu-013 /tmp/syspool

Remove the ZPool Cache File


# cd /tmp/syspool/etc/zfs
# rm -f zpool.cache

Update the Boot Archive


# bootadm update-archive -R /tmp/syspool

Determine the Active Disks


# zpool status syspool

For the sake of this article, let's say the syspool was a three-way mirror and the zpool status returned the following:
  pool: syspool
 state: ONLINE
 scan: resilvered 8.64M in 0h0m with 0 errors on Tue Nov 16 12:34:40 2010
config:
NAME           STATE     READ WRITE CKSUM
        syspool        ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            c6t13d0s0  ONLINE       0     0     0
            c6t14d0s0  ONLINE       0     0     0
            c6t15d0s0  ONLINE       0     0     0

errors: No known data errors

This enumerates the three disk mirror as being composed of disks/slices c6t13d0s0, c6t14d0s0 and c6t15d0s0. We'll use that information for the grub installation.

Install Grub to Each Mirror Disk


# cd /tmp/syspool/boot/grub
# installgrub -f -m stage[12] c6t13d0s0
# installgrub -f -m stage[12] c6t14d0s0
# installgrub -f -m stage[12] c6t15d0s0

Unmount and Reboot


# umount /tmp/syspool
# sync
# reboot

Now, the system should be restored to a bootable configuration based on the selected system checkpoint. A similar procedure can be found on Nexenta's site when using the "Safe Mode" boot option. If you follow that process, you'll quickly encounter an error - likely intentional and meant to elicit a call to support for help. See if you can spot the step...

Monday, November 15, 2010

Short-Take: ZFS version ZPOOL Versions

As features are added to ZFS - the ZFS (filesystem) code may change and/or the underlying ZFS POOL code may change. When features are added, older versions of ZFS/ZPOOL will not be able to take advantage of these new features without the ZFS filesystem and/or pool being updated first.

Since ZFS filesystems exist inside of ZFS pools, the ZFS pool may need to be upgraded before a ZFS filesystem upgrade may take place. For instance, in ZFS pool version 24, support for system attributes was added to ZFS. To allow ZFS filesystems to take advantage of these new attributes, ZFS filesystem version 4 (or higher) is required. The proper order to upgrade would be to bring the ZFS pool up to at least version 24, and then upgrade the ZFS filesystem(s) as needed.

Systems running a newer version of ZFS (pool or filesystem) may "understand" an earlier version. However, older versions of ZFS will not be able to access ZFS streams from newer versions of ZFS.

For NexentaStor users, here are the current versions of the ZFS filesystem (see "zfs upgrade -v"):
VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS filesystem version
 2   Enhanced directory entries
 3   Case insensitive and File system unique identifier (FUID)
 4   userquota, groupquota properties
 5   System attributes

For NexentaStor users, here are the current versions of the ZFS pool (see "zpool upgrade -v"):
VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Deduplication
 22  Received properties
 23  Slim ZIL
 24  System attributes
 25  Improved scrub stats
 26  Improved snapshot deletion performance

As versions change, upgrading the ZFS pool and filesystem is possible using the respective upgrade command. To upgrade all imported ZFS pools, issue the following command as root:
zpool upgrade -a

Likewise, to upgrade the ZFS filesystem(s) inside the pool and all child filesystems, issue the following command as root:
zfs upgrade -r -a

The new ZFS features available to these pool and filesystem version(s) will now be available to the upgraded pools/filesystems.

Friday, September 17, 2010

Quick-Take: ZFS and Early Disk Failure

Anyone who's discussed storage with me knows that I "hate" desktop drives in storage arrays. When using SAS disks as a standard, that's typically a non-issue because there's not typically a distinction between "desktop" and "server" disks in the SAS world. Therefore, you know I'm talking about the other "S" word - SATA. Here's a tale of SATA woe that I've seen repeatedly cause problems for inexperienced ZFS'ers out there...

When volumes fail in ZFS, the "final" indicator is data corruption. Fortunately, ZFS checksums recognize corrupted data and can take action to correct and report the problem. But that's like treating cancer only after you've experienced the symptoms. In fact, the failing disk will likely begin to "under-perform" well before actual "hard" errors show-up as read, write or checksum errors in the ZFS pool. Depending on the reason for "under-performing" this can affect the performance of any controller, pool or enclosure that contains the disk.

Wait - did he say enclosure? Sure. Just like a bad NIC chattering on a loaded network, a bad SATA device can occupy enough of the available service time for a controller or SAS bus (i.e. JBOD enclosure) to make a noticeable performance drop in otherwise "unrelated" ZFS pools. Hence, detection of such events is an important thing. Here's an example of an old WD SATA disk failing as viewed from the NexentaStor "Data Sets" GUI:

[caption id="attachment_1660" align="aligncenter" width="450" caption="Something is wrong with device c5t84d0..."]Disk Statistics showing failing drive[/caption]

Device c5t84d0 is having some serious problems. Busy time is 7x higher than counterparts, and its average service time is 14x higher. As a member of a RAIDz group, the entire group is being held-back by this "under-performing" member. From this snapshot, it appears that NexentaStor is giving us some good information about the disk from the "web GUI" but this assumption would not be correct. In fact, the "web GUI" is only reporting "real time" data so long as the disk is under load. In the case of a lightly loaded zpool, the statistics may not even be reported.

However, from the command shell, historic and real-time access to per-device performance is available. The output of "iostat -exn" shows the count of all errors for devices since the last time counters were reset, and average I/O loads for each:

[caption id="attachment_1662" align="aligncenter" width="450" caption="Device statistics from 'iostat' show error and I/O history."]Device statistics from 'iostat' show error and I/O history.[/caption]

The output of iostat clearly shows this disk has serious hardware problems. It indicates hardware errors as well as transmission errors for the device recognized as 'c5t84d0' and the I/O statistics - chiefly read, write and average service time - implicate this disk as a performance problem for the associated RAIDz group. So, if the device is really failing, shouldn't there be a log report of such an event? Yes, and here's a snip from the message log showing the error:

[caption id="attachment_1663" align="aligncenter" width="450" caption="SCSI error with ioc_status=0x8048 reported in /var/log/messages for failing device."]SCSI error with ioc_status=0x8048 reported in /var/log/messages[/caption]

However, in this case, the log is not "full" with messages of this sort. In fact, it only showed-up under the stress of an iozone benchmark (run from the NexentaStor 'nmc' console). I can (somewhat safely) conclude this to be a device failure since at least one other disk in this group is of the same make, model and firmware revision of the culprit. The interesting aspect about this "failure" is that it does not result in a read, write or checksum error for the associated zpool. Why? Because the device is only loosely coupled to the zpool as a constituent leaf device, and it also implies that the device errors were recoverable by either the drive or the device driver (mapping around a bad/hard error.)

Since these problems are being resolved at the device layer, the ZFS pool is "unaware" of the problem as you can see from the output of 'zpool status' for this volume:

[caption id="attachment_1661" align="aligncenter" width="450" caption="Problems with disk device as yet undetected at the zpool layer."]zpool status output for pool with undetected failing device[/caption]

This doesn't mean that the "consumers" of the zpool's resources are "unaware" of the problem, as the disk error has manifested itself in the zpool as higher delays, lower I/O through-put and subsequently less pool bandwidth. In short, if the error is persistent under load, the drive has a correctable but catastrophic (to performance) problem and will need to be replaced. If, however, the error goes away, it is possible that the device driver has suitably corrected for the problem and the drive can stay in place.

SOLORI's Take: How do we know if the drive needs to be replaced? Time will establish an error rate. In short, running the benchmark again and watching the error counters for the device will determine if the problem persists. Eventually, the errors will either go away or they wont. For me, I'm hoping that the disk fails to give me an excuse to replace the whole pool with a new set of SATA "eco/green" disks for more lab play. Stay tuned...

SOLORI's Take: In all of its flavors, 1.5Gbps, 3Gbps and 6Gbps, I find SATA drives inferior to "similarly" spec'd SAS for just about everything. In my experience, the worst SAS drives I've ever used have been more reliable than most of the SATA drives I've used. That doesn't mean there are "no" good SATA drives, but it means that you really need to work within tighter boundaries when mixing vendors and models in SATA arrays. On top of that, the additional drive port and better typical sustained performance make SAS a clear winner over SATA (IMHO). The big exception to the rule is economy - especially where disk arrays are used for on-line backup - but that's another discussion...

Tuesday, September 14, 2010

Short-Take: iSCSI+Nexenta, Performance Notes

Here are a few performance tips for running iSCSI with NexentaStor in a Windows environment:

  1. When using the Windows iSCSI Software Initiator with some workloads, disabling the Nagle Algorithm on the Windows Server is sometimes recommended;

  2. Tuning TCP window and iSCSI parameters on each side of the connection can deliver better performance;

  3. VMware part of the equation? Adjusting the way VMware handles congestion events could be useful;

  4. On NexentaStor, disable the Nagle Algorithm with a value of "1" (default, 4095, enabled)


For storage applications where latency is a paramount issue, these hints just might help...

Sunday, July 18, 2010

Quick-Take: NexentaStor AD Shares in 100% Virtual SMB

Here's a maintenance note for SMB environments attempting 100% virtualization and relying on SAN-based file shares to simplify backup and storage management: beware the chicken-and-egg scenario on restart before going home to capture much needed Zzz's. If your domain controller is virtualized and it's VMDK file lives on SAN/NAS, you'll need to restart SMB services on the NexentaStor appliance before leaving the building.

Here's the scenario:

  1. An afterhours SAN upgrade in non-HA environment (maybe Auto-CDP for BC/DR, but no active fail-over);

  2. Shutdown of SAN requires shutdown of all dependent VM's, including domain controllers (AD);

  3. End-user and/or maintenance plans are dependent on CIFS shares from SAN;

  4. Authentication of CIFS shares on NexentaStor is AD-based;


Here's the typical maintenance plan (detail omitted):

  1. Ordered shutdown of non-critical VM's (including UpdateManager, vMA, etc.);

  2. Ordered shutdown of application VM's;

  3. Ordered shutdown of resource VM's;

  4. Ordered shutdown of AD server VM's (minus one, see step 7);

  5. Migrate/vMotion remaining AD server and vCenter to a single ESX host;

  6. Ordered shutdown of ESX hosts (minus one, see step 8);

  7. vSphere Client: Log-out of vCenter;

  8. vSphere Client: Log-in to remaining ESX host;

  9. Ordered shutdown of vCenter;

  10. Ordered shutdown of remaining AD server;

  11. Ordered shutdown of remaining ESX host;

  12. Update SAN;

  13. Reboot SAN to update checkpoint;

  14. Test SAN update - restoring previous checkpoint if necessary;

  15. Power-on ESX host containing vCenter and AD server (see step 8);

  16. vSphere Client: Log-in to remaining ESX host;

  17. Power-on AD server (through to VMware Tools OK);

  18. Restart SMB service on NexentaStor;

  19. Power-on vCenter;

  20. vSphere Client: Log-in to vCenter;

  21. vSphere Client: Log-out of ESX host;

  22. Power-on remaining ESX hosts;

  23. Ordered power-on of remaining VM's;


A couple of things to note in an AD environment:

  1. NexnetaStor requires the use of AD-based DNS for AD integration;

  2. AD-based DNS will not be available at SAN re-boot if all DNS servers are virtual and only one SAN is involved;

  3. Lack of DNS resolution on re-boot will cause a failure for DNS name based NTP service synchronization;

  4. NexentaStor SMB service will fail to properly initialize AD credentials;

  5. VMware 4.1 now pushes AD authentication all the way to ESX hosts, enabling better credential management and security but creating a potential AD dependency as well;

  6. Using auto-startup order on the remaining ESX host for AD and vCenter could automate the process (steps 17 & 19), however, I prefer the "manual" approach after a SAN upgrade in case the upgrade failure is detected only after ESX host is restarted (i.e. storage service interaction in NFS/iSCSI after upgrade).


SOLORI's Take: This is a great opportunity to re-think storage resources in the SMB as the linchpin to 100% virtualization.  Since most SMB's will have a tier-2 or backup NAS/SAN (auto-sync or auto-CDP) for off-rack backup, leveraging a shared LUN/volume from that SAN/NAS for a backup domain controller is a smart move. Since tier-2 SAN's may not have the IOPs to run ALL mission critical applications during the maintenance interval, the presence of at least one valid AD server will promote a quicker RTO, post-maintenance, than coming up cold. [This even works with DAS on the ESX host]. Solution - add the following and you can ignore step 15:

3a. Migrate always-on AD server to LUN/volume on tier-2 SAN/NAS;


24. Migrate always-on AD server from LUN/volume on tier-2 SAN/NAS back to tier-1;


Since even vSphere Essentials Plus has vMotion now (a much requested and timely addition) collapsing all remaining VM's to a single ESX host is a no brainer. However, migrating the storage is another issue which cannot be resolved without either a shutdown of the VM (off-line storage migration) or Enterprise/Enterprise Plus version of vSphere. That is why the migration of the AD server from tier-2 is reserved for last (step 17) - it will likely need to be shutdown to migrate the storage between SAN/NAS appliances.

Thursday, July 15, 2010

ZFS Pool Import Fails After Power Outage

The early summer storms have taken its toll on Alabama and UPS failures (and short-falls) have been popping-up all over. Add consolidated, shared storage to the equation and you have a recipe for potential data loss - at least this is what we've been seeing recently. Add JBOD's with separate power rails and limited UPS life-time and/or no generator backup and you've got a recipe for potential data loss.



Even with ZFS pools, data integrity in a power event cannot be guaranteed - especially when employing "desktop" drives and RAID controllers with RAM cache and no BBU (or perhaps a "bad storage admin" that has managed to disable the ZIL). When this happens, NexentaStor (an other ZFS storage devices) may even show all members in the ZFS pool as "ONLINE" as if they are awaiting proper import. However, when an import is attempted (either automatically on reboot or manually) the pool fails to import.




From the command line, the suspect pool's status might look like this:


root@NexentaStor:~# zpool import
pool: pool0
id: 710683863402427473
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
        pool0        ONLINE
          mirror-0   ONLINE
            c1t12d0  ONLINE
            c1t13d0  ONLINE
          mirror-1   ONLINE
            c1t14d0  ONLINE
            c1t15d0  ONLINE

Looks good, but the import it may fail like this:

root@NexentaStor:~# zpool import pool0
cannot import 'pool0': I/O error

Not good. This probably indicates that something is not right with the array. Let's try to force the import and see what happens:



Nope. Now this is the point where most people start to get nervous, their neck tightens-up a bit and they begin to flip through a mental calendar of backup schedules and catalog backup repositories - I know I do. However, it's the next one that makes most administrators really nervous when trying to "force" the import:



root@NexentaStor:~# zpool import -f pool0
pool: pool0
id: 710683863402427473
status: The pool metadata is corrupted and the pool cannot be opened.
action: Restore the file in question if possible.  Otherwise restore the
entire pool from backup.
cannot import 'pool0': I/O error


Really not good. Did it really suggest going to backup? Ouch!.



In this case, something must have happened to corrupt metadata - perhaps the non-BBU cache on the RAID device when power failed. Expensive lesson learned? Not yet. The ZFS file system still presents you with options, namely "acceptable data loss" for the period of time accounted for in the RAID controller's cache. Since ZFS writes data in transaction groups and transaction groups normally commit in 20-30 second intervals, that RAID controller's lack of BBU puts some or all of that pending group at risk. Here's how to tell by testing the forced import as if data loss was allowed:



root@NexentaStor:~# zpool import -nfF pool0
Would be able to return data to its state as of Fri May 7 10:14:32 2010.
Would discard approximately 30 seconds of transactions.


or

root@NexentaStor:~# zpool import -nfF pool0
WARNING: can't open objset for pool0

If the first output is acceptable, then proceeding without the "n" option will produce the desired effect by "rewinding" the last couple of transaction groups (read ignoring) and imported the "truncated" pool. The "import" option will report the exact number of "seconds" worth of data that cannot be restored. Depending on the bandwidth and utilization of your system, this could be very little data or several MB worth of transaction(s).



What to do about the second option? From the man pages on "zpool import" Sun/Oracle says the following:



zpool import [-o mntopts] [ -o property=value] ... [-d dir | -c cachefile] [-D] [-f] [-R root] [-F [-n]]-a
Imports all pools found in the search directories. Identical to the previous command, except that all pools with a sufficient number of devices available are imported. Destroyed pools, pools that were previously destroyed with the “zpool destroy” command, will not be imported unless the-D option is specified.



-o mntopts
Comma-separated list of mount options to use when mounting datasets within the pool. See zfs(1M) for a description of dataset properties and mount options.

-o property=value
Sets the specified property on the imported pool. See the “Properties” section for more information on the available pool properties.

-c cachefile
Reads configuration from the given cachefile that was created with the “cachefile” pool property. This cachefile is used instead of searching for devices.

-d dir
Searches for devices or files in dir. The -d option can be specified multiple times. This option is incompatible with the -c option.

-D
Imports destroyed pools only. The -f option is also required.

-f
Forces import, even if the pool appears to be potentially active.

-F
Recovery mode for a non-importable pool. Attempt to return the pool to an importable state by discarding the last few transactions. Not all damaged pools can be recovered by using this option. If successful, the data from the discarded transactions is irretrievably lost. This option is ignored if the pool is importable or already imported.

-a
Searches for and imports all pools found.

-R root
Sets the “cachefile” property to “none” and the “altroot” property to “root”.

-n
Used with the -F recovery option. Determines whether a non-importable pool can be made importable again, but does not actually perform the pool recovery. For more details about pool recovery mode, see the -F option, above.


No real help here. What the documentation omits is the "-X" option. This option is only valid with the "-F" recovery mode setting, however it is NOT well documented suffice to say it is the last resort before acquiescing to real problem solving... Assuming the standard recovery mode "depth" of transaction replay is not quite enough to get you over the hump, the "-X" option gives you an "extended replay" by seemingly providing a scrub-like search through the transaction groups (read "potentially time consuming") until it arrives at the last reliable transaction group in the dataset.

Lessons to be learned from this excursion into pool recovery are as follows:



  1. Enterprise SAS good; desktop SATA could be a trap

  2. Redundant Power + UPS + Generator = Protected; Anything else = Risk

  3. SAS/RAID Controller + Cache + BBU = Fast; SAS/RAID Controller + Cache - BBU = Train Wreck



The data integrity functions in ZFS are solid when used appropriately. When architecting your HOME/SOHO/SMB NAS appliance, pay attention to the hidden risks of "promised performance" that may walk you down the plank towards a tape backup (or resume writing) event. Better to leave the 5-15% performance benefit on the table or purchase adequate BBU/UPS/Generator resources to supplant your system in worst-case events. In complex environments, a pending power loss can be properly mitigated through management supervisors and clever scripts: turning down resources in advance of total failure. How valuable is your data???

Thursday, June 17, 2010

In-the-Lab: Install VMware Tools on NexentaStor VSA

Physical lab resources can be a challenge to "free-up" just to test a potential storage appliance. With NexentaStor, you can download a pre-configured VMware (or Xen) appliance from NexentaStor.Org, but what if you want to build your own? Here's a little help on the subject:

  1. Download the ISO from NexentaStor.Org (see link above);

  2. Create a VMware virtual machine:

    1. 2 vCPU

    2. 4GB RAM (leaves about 3GB for ARC);

    3. CD-ROM (mapped to the ISO image);

    4. One (optionally two if you want to simulate the OS mirror) 4GB, thin provisioned SCSI disks (LSI Logic Parallel);

    5. Guest Operating System type: Sun Solaris 10 (64-bit)

    6. One E1000 for Management/NAS

    7. (optional) One E1000 for iSCSI



  3. Streamline the guest by disabling unnecessary components:

    1. floppy disk

    2. floppy controller (remove from BIOS)

    3. primary IDE controller (remove from BIOS)

    4. COM ports (remove from BIOS)

    5. Parallel ports (remove from BIOS)



  4. Boot to ISO and install NexentaStor CE

    1. (optionally) choose second disk as OS mirror during install



  5. Register your installation with Nexenta

    1. http://www.nexenta.com/register-eval

    2. (optional) Select "Solori" as the partner



  6. Complete initial WebGUI configuration wizard

    1. If you will join it to a domain, use the domain FQDN (i.e. microsoft.msft)

    2. If you choose "Optimize I/O performance..." remember to re-enable ZFS intent logging under Settings>Preferences>System

      1. Sys_zil_disable = No





  7. Shutdown the VSA

    1. Settings>Appliance>PowerOff



  8. Re-direcect the CD-ROM

    1. Connect to Client Device



  9. Power-on the VSA and install VMware Tools

    1. login as admin

      1. assume root shell with "su" and root password



    2. From vSphere Client, initiate the VMware Tools install

    3. cd /tmp

      1. untar VMware Tools with "tar zxvf  /media/VMware\ Tools/vmware-solaris-tools.tar.gz"



    4. cd to /tmp/vmware-tools-distrib

      1. install VMware Tools with "./vmware-install.pl"

      2. Answer with defaults during install



    5. Check that VMware Tools shows and OK status

      1. IP address(es) of interfaces should now be registered

        [caption id="attachment_1580" align="aligncenter" width="300" caption="VMware Tools are registered."][/caption]





  10. Perform a test "Shutdown" of your VSA

    1. From the vSphere Client, issue VM>Power>Shutdown Guest

      [caption id="attachment_1581" align="aligncenter" width="300" caption="System shutting down from VMware Tools request. "][/caption]

    2. Restart the VSA...

      [caption id="attachment_1582" align="aligncenter" width="300" caption="VSA restarting in vSphere "][/caption]




Now VMware Tools has been installed and you're ready to add more virtual disks and build ZFS storage pools. If you get a warning about HGFS not loading properly at boot time:

[caption id="attachment_1586" align="aligncenter" width="300" caption="HGFS module mismatch warning."][/caption]

it is not usually a big deal, but the VMware Host-Guest File System (HGFS) has been known to cause issues in some installations. SInce the NexentaStor appliance is not a general purpose operating system, you should customize the install to not use HGFS at all. To disable it, perform the following:

  1. Edit "/kernel/drv/vmhgfs.conf"

    1. Change:     name="vmhgfs" parent="pseudo" instance=0;

    2. To:     #name="vmhgfs" parent="pseudo" instance=0;



  2. Re-boot the VSA


Upon reboot, there will be no complaint about the offending HGFS module. Remember that, after updating VMware Tools at a future date, the HGFS configuration file will need to be adjusted again. By the way, this process works just as well on the NexentaStor Commercial edition, however you might want to check with technical support prior to making such changes to a licensed/supported deployment.

Monday, June 14, 2010

NexentaStor CIFS Shares with Active Directory Authentication

Sharing folders in NexentaStor is pretty easy in Workgroup mode, but Active Directory integration takes a few extra steps.  Unfortunately, it's not (yet) as easy as point-and-click, but it doesn't have to be too difficult either. (The following assumes/requires that the NexentaStor appliance has been correctly configured-in and joined-to Active Directory.)

[caption id="attachment_1562" align="alignright" width="170" caption="Typical user and group permissions for a local hard disk in Windows."][/caption]

Let's examine the case where a domain admin group will have "Full Control" of the share, and "Everyone" will have read/execute permissions. This is a typical use case where a single share contains multiple user directories under administrative control. It's the same configuration as local disks in a Windows environment. For our example, we're going to mimic this setup using a CIFS share from a NexentaStor CE appliance and create the basic ACL to allow for Windows AD control.

For this process to work, we need to join the NexentaStor appliance to the Active Directory Domain. The best practice is to create the machine account in AD first, assign control user/group rights (if possible) and then attempt to join. It is IMPORTANT that the host name and DNS configuration of the NexentaStor appliance match domain norms, or things will come crashing to a halt pretty quickly.

That said, assuming that your DC is 1.1.1.1 and your BDC is 1.1.1.2 with a "short" domain of "SOLORI" and a FQDN of "SOLORI.MSFT" your NexentaStor's name server configuration (Settings->Network->Name Servers) would look something like this:

This is important because the AD queries will pull service records from the configured domain name servers. If these point to an "Internet" DNS server, the AD entries may not be reflected in that server's database and AD authentication (as well as join) will fail.

The other way the NexentaStor appliance knows what AD Domain to look into is by its own host name. For AD authentication to work properly, the NexentaStor host name must reflect the AD domain. For example, if the FQDN of your AD domain is "SOLORI.MSFT" then your domain name on the appliance would be configured like this (Appliance->Basic Settings->Domainname):

The next step is to create the machine account in AD using "Active Directory Users and Computers" administrator's configuration tool. Find your domain folder and right-click "Computers" - select New->Computer from the menu and enter the computer name (no domain). The default user group assigned to administrative control should be Domain Admins. Since this works for our example, no changes are necessary so click "OK" to complete.

Now it's time to join the AD domain from NexentaStor. Any user with permissions to join a machine to the domain will do. Armed with that information, drill down to Data Management->Shares->CIFS Server->Join AD/DNS Server and enter the AD/DNS server. AD server, AD user and user password into the configuration box:

If your permissions and credentials are good, your NexentaStor appliance is not a member of your domain. As such, it can now identify AD users and groups by unique gid and uid data created from AD. This gid and uid information will be used to create our ACLs for the CIFS share.

To uncover the gid for the "Domain Admins" and "Domain Users" groups, we issue the following from the NexentaStor NMC (CLI):
nmc@san01:/$ idmap dump -n | grep "Domain Admins"

wingroup:Domain Admins@solori.msft     ==      gid:3036392745


nmc@san01:/$ idmap dump -n | grep "Domain Users"

wingroup:Domain Users@solori.msft     ==      gid:1238392562

Now we can construct a CIFS share (with anonymous read/write disabled) and apply the Domain Admin gid to an ACL - just click on the share, and then click "(+) Add Permissions for Group":

[caption id="attachment_1568" align="aligncenter" width="300" caption="Applying administrative permissions with the AD group ID for Domain Admins."][/caption]

We do similarly with the Domain User gid:

[caption id="attachment_1569" align="aligncenter" width="300" caption="Applying the Domain User gid to CIFS share ACL."][/caption]

Note that the "Domain Users" group gets only "execute" and "read" permissions while the "Domain Admins" group gets full control - just like the local disk! Now, with CIFS sharing enabled and the ACL suited to our AD authentication, we can access the share from any domain machine provided our user is in the Domain Users or Admins group.

Administrators can now create "personal" folders and assign detailed user rights just as they would do with any shared storage device. The only trick is in creating the initial ACL for the CIFS share - as about - and you've successfully integrated your NexentaStor appliance into you AD domain.

NOTE: If you're running Windows Server 2008 (or SBS 2008) as your AD controller, you will need to update the share mode prior to joining the domain using the following command (from root CLI):
# sharectl set -p lmauth_level=2 smb

NOTE: I’ve also noticed that, upon reboot of the appliance (i.e. after a major update of the kernel/modules) your ephemeral id mapping takes some time to populate during which time authentication failures to CIFS shares can fail. This appears to have something to do with the state of ephemeral-to-SID mapping after re-boot.

To enable the mapping of unresolvable SIDs, do the following:
$ svccfg -s idmap setprop config/unresolvable_sid_mapping = boolean: true
$ svcadm refresh idmap

Tuesday, March 2, 2010

NexentaStor 3.0 Announced

Nexenta Systems announced it's 3.0 iteration at CeBIT and in a press release this week and has provided a few more details about how the next version is shaping-up. Along with the previously announced deduplication features, the NexentaStor 3.0 edition will include several enhancements to accelerate performance and virtualization applications:

  • In-line deduplication for increased storage savings (virtual machine templates, clones, etc.);

  • Broader support for 10GE adapters and SAS-2 (6Gbps, zoning, etc.) adapters;

  • Replication enhancements to simplify disaster recovery implementations;

  • An updated Virtual Machine Data Center (VMDC v3.0) optional plug-in with VMware, Xen and Hyper-V support (storage-centric control of virtual machine resource provisioning and management);


Additionally, Nexenta is promising easier high-availability (Simple and HA cluster) provisioning for mission critical implementations. Existing NexentaStor license holders will be able to upgrade to NexentaStor 3.0 at no additional cost. Nexenta Systems plans to make NexentaStor 3.0 available by the end of March 2010.

As a Nexenta partner, Solution Oriented will provide clients with valid NexentaStor support contracts upgrade guidance once NexentaStor 3.0 has been released and tested against SOLORI stable image storage platforms. As always, Nexenta's VMware Ready virtual storage appliance should be your first step in evaluating upgrade potential.

Tuesday, November 3, 2009

Sun Adds De-Duplication to ZFS

Yesterday Jeff Bonwick (Sun) announced that deduplication is now officially part of ZFS - Sun's Zettabyte File System that is at the heart of Sun's Unified Storage platform and NexentaStor. In his post, Jeff touched on the major issues surrounding deduplication in ZFS:
Deduplication in ZFS is Block-level

ZFS provides block-level deduplication because this is the finest granularity that makes sense for a general-purpose storage system. Block-level dedup also maps naturally to ZFS's 256-bit block checksums, which provide unique block signatures for all blocks in a storage pool as long as the checksum function is cryptographically strong (e.g. SHA256).

Deduplication in ZFS is Synchronous

ZFS assumes a highly multithreaded operating system (Solaris) and a hardware environment in which CPU cycles (GHz times cores times sockets) are proliferating much faster than I/O. This has been the general trend for the last twenty years, and the underlying physics suggests that it will continue.

Deduplication in ZFS is Per-Dataset

Like all zfs properties, the 'dedup' property follows the usual rules for ZFS dataset property inheritance. Thus, even though deduplication has pool-wide scope, you can opt in or opt out on a per-dataset basis. Most storage environments contain a mix of data that is mostly unique and data that is mostly replicated. ZFS deduplication is per-dataset, which means you can selectively enable dedup only where it is likely to help.

Deduplication in ZFS is based on a SHA256 Hash

Chunks of data -- files, blocks, or byte ranges -- are checksummed using some hash function that uniquely identifies data with very high probability. When using a secure hash like SHA256, the probability of a hash collision is about 2^-256 = 10^-77. For reference, this is 50 orders of magnitude less likely than an undetected, uncorrected ECC memory error on the most reliable hardware you can buy.

Deduplication in ZFS can be Verified

[If you are paranoid about potential "hash collisions"] ZFS provies a 'verify' option that performs a full comparison of every incoming block with any alleged duplicate to ensure that they really are the same, and ZFS resolves the conflict if not.

Deduplication in ZFS is Scalable

ZFS places no restrictions on your ability to dedup. You can dedup a petabyte if you're so inclined. The performace of ZFS dedup will follow the obvious trajectory: it will be fastest when the DDTs (dedup tables) fit in memory, a little slower when they spill over into the L2ARC, and much slower when they have to be read from disk -- but the point I want to emphasize here is that there are no limits in ZFS dedup. ZFS dedup scales to any capacity on any platform, even a laptop; it just goes faster as you give it more hardware.

Jeff Bonwick's Blog, November 2, 2009



What does this mean for ZFS users? That depends on the application, but highly duplicated environments like virtualization stand to gain significant storage-related value from this small addition to ZFS. Considering the various ways virtualization administrators deal with virtual machine cloning, even the basic VMware template approach (not using linked-clones) will now result in significant storage savings. This restores parity between storage and compute in the virtualization stack.


What does it mean for ZFS-based storage vendors? More main memory and processor threads will be necessary to limit the impact on performance. With 6-core and 8-thread CPU's available in the mainstream, this problem is very easily resolved. Just like the L2ARC tables consume main memory, the DDT's will require an increase in main memory for larger datasets. Testing and configuration convergence will likely take 2-3 months once dedupe is mainstream.


When can we expect to see dedupe added to ZFS (i.e. OpenSolaris)? According to Jeff, "in roughly a month."


Updated: 11/04/2009 - Link to Nexenta corrected. Was incorrectly linked to "nexent.com" - typo - now correctly linked to "http://www.nexenta.com"


Tuesday, June 30, 2009

Nexenta Turns 2.0

Beth Pariseau, Senior News Writer for SearchStorage.com, has an interview with Nexenta CEO Evan Powell about the release of NexentaStor 2.0 today. The open source storage vendor is making some fundamentally "enterprise focused" changes to its platform in this release by adding active-active high availability features and 24/7 phone support.
"Version 2.0 is Nexenta's attempt to "cross the chasm" between the open-source community and the traditional enterprise. Chief among these new features is the ability to perform fully automated two-way high availability between ZFS server nodes. Nexenta has already made synchronous replication and manual failover available for ZFS, which doesn't offer those features natively, Powell said. With the release of Nexenta's High Availability 1.0 software, failover and failback to the secondary server can happen without human intervention."

- SearchStorage.com



In a related webinar and conference call today, Powell reiterated Nexenta's support of open storage saying, "we believe that you should own your storage. Legacy vendors want to lock you into their storage platform, but with Nexenta you can take your storage to any platform that speaks ZFS." Powell sees Nexenta's anti-lock-in approach as part of their wider value proposition. When asked about de-duplication technology, he referred to Sun's prototyped de-duplication technology and the promise to introduce it into the main line this summer.

Wednesday, May 6, 2009

Add SSD to Your ZIL

[caption id="" align="alignright" width="200" caption="Samsung's new SSD generation using multi-level cell (MLC) flash and a multi-channel flash controller with NCQ and 128MB SDRAM cache."]Samsungs new SSD generation using multi-level cell (MLC) flash and a multi-channel flash controller with NCQ and 128MB SDRAM cache.[/caption]

Tom's Hardware has a good review on the state of current SSD options out there. As discussed in previous posts, the ZFS file system offers hybrid storage aspects out of the box. This game-changing technology allows for "holy grail" levels of price-performance with the key technology being SSD for caching. That's the value proposition our friends at Nexenta have been preaching.

To see what this means in a ZFS storage environment, go no farther than Sun's blog: Brendan Gregg has posted a great blog on how ZFS' L2ARC can be comitted to SSD to dramatically increase effective IOPS and drastically reduce latency. The results speak for themselves...

Tuesday, April 21, 2009

Sun Finds a Buyer in Oracle

Sun and Oracle have come to terms on a $7.4B cash deal. Oracle's Ellison rejected a similar deal in 2003 due to bad timing and a PeopleSoft acquisition. Says Sun's post:

"Sun and Oracle today announced a definitive agreement for Oracle to acquire Sun for $9.50 per share in cash. The Sun Board of Directors has unanimously approved the transaction. It is anticipated to close this summer."


"The acquisition of Sun transforms the IT industry, combining best-in-class enterprise software and mission-critical computing systems," said Oracle CEO Larry Ellison. "Oracle will be the only company that can engineer an integrated system - applications to disk - where all the pieces fit and work together so customers do not have to do it themselves. Our customers benefit as their systems integration costs go down while system performance, reliability and security go up."



Oracle's press release mirror's Sun's:

Tuesday, March 17, 2009

In-the-Lab: VSA Shoot-out

We're cooking with gas over the next two weeks, making performance contrasts using similar resource build-outs of:

  1. FreeNAS (0.7 nightly build)

  2. OpenFiler 2.3

  3. NexentaStor (1.1.5 dev)

  4. SunStorage (VSA)


All configured as VSA's under ESXi with pass-thru access to local attached storage. Why do this? ESX provides some supervisory analysis that makes quick lab analysis easy. Also, VSA storage is a 2009 killer-app and absolutely necessary for targeted micro-deployments.

These are not indepth storage studys, but an out-of-the-box analysis of what to expect and where to spend your time=money. Stay tuned...

Thursday, February 26, 2009

Search Term Review: February 2009

Looking at the "Top 3" search index  driving traffic to our blog, its obvious that ESXi and FreeNAS are getting a lot of attention.

Search Term #1: esxi network drivers


White box and DIY lab explorers are trying to push the envelope on network cards. Except for supported-but-PCI-id-does-not-match exceptions, it is not worthwhile to hack-in a "cheap" NIC for your hypervisor. Since NAS/SAN and application data traverses your "hacked NIC," do you really want to risk it? For the exception, see the next search term.

Search Term #2: nvidia mcp55 sata controller esxi install


The number of MCP55-based motherboards out there is understandably large. However, not all MCP55 systems are "install" compatible. That's where USB flash install makes your ESXi installation an easy - albeit two step - process. There are blog entries out there that will help you on your way to MCP55/ESX/ESXi nirvana.

The basic problem is your controller has a different revision - and PCI id - then the driver expects. Since this "compatible id list" was cooked into the ISO image, it is likely stale and your "version" of the hardware "may" work. This is common to MCP and ICH SATA controllers. If so, modifying the list of supported PCI ids and making the correct driver associations is the trick to success.

Here's a good blog outlining the hard way for ESX.

Hers another on creating your own roll-up for ESXi. It includes a downloadable oem.tgz replacement with support for ICH10, ICH8, e1000e, 3Ware, igb and Dvorak support. The discussion give additional insight.

Search Term #3: freenas usb


This is an easy one. We have decent how-to here, and there are numerous others on-line. SOLORI has not explored the nightly build (AMD64) variant - yet. If you want to explore the "wild side" of FreeNAS, go here, the FreeNAS community is vibrant and informative.

Tuesday, February 17, 2009

EMC Celerra VSA

Virtual Storage Appliances (VSA) are really picking-up these days. Even the legacy hardware vendors are getting into the fray. They're great in the lab and - assuming tailored performance - can fit some really interesting "embedded" applications.

If you haven't had a chance to look at the EMC Celerra VSA then click over to Chad Sakac's VirtualGeek Blog and have a look at his series on configuration and testing. If you have time, check out some of Chad's more visionary stuff while your there - he has some good information on what's to come in the next few months from EMC and VMware...

For the Celerra VSA, look at these two links

Tuesday, February 10, 2009

Installing: Xtravirt Virtual SAN

Today we're looking at the Xtravirt Virtual SAN Appliance (VSA) solution for use with VMware ESX Server 3. It is designed to be a simple to deploy, redundant (DRBD synchronization), high-availability iSCSI SAN between two ESX servers. We are installing it on two ESXi servers, each with local storage, running the latest patch update (3.5.0 build 143129).

Initial Installation


XVS requires a virtual machine import: either using Converter or manual process. We follow the manual process. We used the command line to convert the imported disk into a ESXi-compliant format:
vmkfstools -i XVS.vmdk -d thin XVSnew.vmdk
rm -f XVS.vmdk XVS-*

After conversion, you have a 2GB virtual machine (times two) ready for configuration. We removed the legacy ethernet and hard disk that came with the inventory import. Then add the "existing" disk and new Ethernet (flex) controller.

We then added a 120GB virtual disk to each node using the local storage controllers: LSI 1068SAS (RAID1) for node 1 and NVidia MCP55Pro (RAID1) for node 2. Node 1 and 2 are using the same 250GB Seagate ES.2 (RAID edition) drives.

Xtravirt Virtual SAN Appliance

The gurus at Xtravirt have cooked-up a virtual SAN appliance (VSA) to be run directly on ESX using local VMFS storage as the SAN backing. While this approach may imply some performance hits, the price is worth a look: free.

The appliance is designed to be deployed on two ESX hosts and replicates synchronously between them for fail-over capability. We'll throw their VSA onto our lab machines and see what the fuss is about.

Sunday, February 8, 2009

Update: FreeNAS and ESX 3i

NOTE: There seems to be a known problem with local pass-through RDM and LSI Logic SCSI controllers related to this post. FreeNAS running on ESX Server 3i using virtual disks from VMFS storage does not exhibit a failure problem, although the logging issue remains.

FreeNAS (0.69) and VMware ESX Server 3i (ESXi) would seem like a good "free as in speech" pairing for low-end virtualization. Unfortunately, the reality is not a good one. Mine's not the first problem with FreeNAS and VMware.

FreeNAS's iSCSI target and VMware ESX Server 3i (update 3) get along fine until you push the connection between them. Initially, ESXi finds the target, formats it and appears to be working fine: you can even create some light-weight VM's on the allocated storage. However, it's when copying VM's from one storage array to another where things go wrong.