Update: FreeNAS and ESX 3i

NOTE: There seems to be a known problem with local pass-through RDM and LSI Logic SCSI controllers related to this post. FreeNAS running on ESX Server 3i using virtual disks from VMFS storage does not exhibit a failure problem, although the logging issue remains.

FreeNAS (0.69) and VMware ESX Server 3i (ESXi) would seem like a good "free as in speech" pairing for low-end virtualization. Unfortunately, the reality is not a good one. Mine's not the first problem with FreeNAS and VMware.

FreeNAS's iSCSI target and VMware ESX Server 3i (update 3) get along fine until you push the connection between them. Initially, ESXi finds the target, formats it and appears to be working fine: you can even create some light-weight VM's on the allocated storage. However, it's when copying VM's from one storage array to another where things go wrong.

Once FreeNAS (embedded, AMD64 version) logs several thousand page 85 errors, the session drops between ESXi and the storage is marked bad, the filesystem corrupts and requires reformatting before ESXi will accept the target for use again. This renders the target unacceptable for even the testing of VMware ESX Server 3i.

Oddly, the performance of FreeNAS under VMware (i.e. virtualized using "local storage" from ESXi) seems to be slow (about 5-8MB/sec writes) but usable. The "failure" did not occur however the incessant error messages continued. This might lead one to deduce a hardware issue with FreeNAS on bare metal, and I am willing to accept that for now, suffice to say iSCSI targets must be 100% reliable or they are not usable as a foundation for virtualization.

System: Tyan 2935 (BIOS v1.03), Opteron 2376, 16GB DDR2-667, LSI3442E SAS/SATA (SAS1068E), 4x250GB drives.

Note, this is NOT a problem with using FreeNAS as an NFS platform in conjunction with ESXi: the problem appears to be with the iSCSI target in FreeNAS and ESX 3i's expectations. This does seem to be an interaction with caching either from VMware's perspective or from FreeNAS's perspective. I will follow-up with some regression tests in a few weeks.

Updated: 10-Feb-2009