illallangi.com

Woke up this morning to the sound of my desktop computer starting up. Thought it was a little weird as my nothing should be rebooting it, my recent wins in ConfigMgr means that no updates should have been applied, and the computer was left on overnight.

Turns out it was a power failure, which is never a good thing in my house as my server farm currently is not UPS protected. Normally means I have to manually start up my windows environment, hosted on two ESXi hosts connecting to shared storage on a Solaris 11 box. However this morning upon connecting to each ESXi box no storage was showing up. SSHing to my SolarisBox showed that two of my drives were unavailable. Both of these drives were in the same array, so that array had not come online. That array holds all the virtual servers from my ESXi servers (the bulk of my data is on the other array).

Fortunately power cycling the box bought the drives back up, with the array in a degraded state. While all four drives were showing as available, the array had encountered errors in every exported LUN, so every one of my virtual servers would now be suspect.

Drove to the markets to buy replacement disks, as I no longer could trust these ones. The old disks had a reasonable life, I’d bought them at various times during my first two years in Canberra so they were about five years old by this point. Ended up with 4x Western Digital WD30EZRX disks, each 3TB in size. Seemed like a good deal to me, and should see available disk space (after redundancy and formatting loss) increasing from 900GB to 10TB.

Turns out the WD 3.0TB drives use a feature called “Advanced Formatting”, which is not compatible with Solaris. Once drives cross the 2.19TB size the on disk clusters must increase in size from 512 bytes to 4 kilobytes. As the majority of operating systems out there get confused with disks that have clusters that big WD has hidden this fact by configuring their larger drives to report 512 byte clusters.

This is normally not an issue, however ZFS on Solaris works a lot closer with the disk than consumer operating systems, and needs to know the correct cluster size. A mismatch in the disks actual cluster size and the size that Solaris is using for a ZFS Pool can cause a large performance loss. You cannot currently override the automatically detected cluster size Solaris uses.

In order to overcome this limitation I had to create the initial pool with a FreeBSD Live CD. I initially used ZFSguru to achieve this, as it had a nice web interface that provides a tick box to just do that, however Solaris would refuse to import the created pools properly. I was receiving Primary Label Corrupt errors and two of the four disks in the pool were showing in the import command as partitions, rather than the whole disk. I ended up downloading a patched version of the ZPool binary and using it instead.

With a recent clean up of my websites, it may be time to rebuild my web server. Illallangi.com is hosted on a virtual server provided by Linode, who operate data centres in several locations.

With their recent opening of a data center in the Asia-Pacific region (Tokyo, Japan), I’ve been questioning my current servers location in Newark.

As most of my users are located in Australia, with the majority of people hosted on my service being users of Internode, I’m aiming to have the lowest latency and fastest download speed for users on Internode’s network.

In order to determine which of Linode’s six locations would be most appropriate for use, I first downloaded the six speed test files to my Solaris server to see how they performed. I used my Solaris server as it was a convenient box with a copy of wget; the same tests could have been performed using any download tool. The files are 100 megabytes each, one on a server in each of Linode’s locations.

Continue reading Selecting a server location