The Home Server Part 3: ZFS Recovery

To illustrate zfs recovery, and to check it works. I am going to:

  1. Create a number of files, of various sizes.
  2. Produce a strong hash of those files sha512
  3. Shutdown
  4. Remove a disk
  5. Put in a different disk, but of the same size
  6. Repair the pool with the new disk – somehow
  7. Validate that the files I created, match the recovered files by checking their hashes

Now this is only a basic test, and wont cover all scenarios. Its more about getting acquainted with zfs, checking it works as I hope (although the large body of users using it, hopefully means it does). And learning how to rebuild my pool in the case of a disk failure.

The latter is probably the most important part.

So lets create some files. I just ran the command below a number of times changing the output filename and count to change the size of the resulting file.

dd if=/dev/urandom of=testfile1.a bs=1M count=1024

As a interesting note, I saw I was getting a consistent 10MB/s throughput with this which seems quite good.

Next I did

ls -al > ~/test_ls_output.txt

To create a directory listing I can use later for comparison.

Then I generated the hashes:

sha512sum testfile1.a > ~/testfile1.a-CHECKSUM

Then to validate the hash:

sha512sum -c ~/testfile1.a-CHECKSUM

So that’s steps 1 and 2 done, now to shutdown and switch out the drives.

So having rebooted I ran:

$> zpool status

pool: storage


status: One or more devices could not be used because the label is missing or         invalid.  Sufficient replicas exist for the pool to continue         functioning in a degraded state. action: Replace the device using ‘zpool replace’.    see:

scan: none requested config:

NAME                                           STATE     READ WRITE CKSUM

storage                                        DEGRADED     0     0     0

raidz1-0                                     DEGRADED     0     0     0

ata-WDC_WD5000AAKS-00TMA0_WD-WCAPW1902486  ONLINE       0     0    0

ata-WDC_WD5000AAKS-00TMA0_WD-WCAPW1908191  ONLINE       0     0     0

ata-SAMSUNG_HD501LJ_S0MUJ13P651726         UNAVAIL      0     0     0

errors: No known data errors

So we can see a disk is missing, and the pool is degraded (its raid5/RAIDZ so we can still access the pool and its data!) And it gives a hint on how to fix it.

A quick google, and the following command tells zfs to switch out the old disk, with the new one, and to then resilver. (I had to use -f to force it too, as my replacement disk had data and other bits on)

zpool replace storage -f /dev/disk/by-id/ata-SAMSUNG_HD501LJ_S0MUJ13P651726 /dev/disk/by-id/ata-SAMSUNG_HD501LJ_S0MUJ13P651727

If you run

$> zpool status

(only interesting parts of output included)

scan: resilver in progress since Sun Feb 23 10:19:11 2014
265M scanned out of 5.25G at 33.1M/s, 0h2m to go
86.2M resilvered, 4.92% done

replacing-2                                UNAVAIL      0     0     0
ata-SAMSUNG_HD501LJ_S0MUJ13P651726       UNAVAIL      0     0     0
ata-SAMSUNG_HD501LJ_S0MUJ13P651727       ONLINE       0     0     0  (resilvering)

About 3minutes later it finished, and running zpool status again, shows the zpool all backup and running again fine.

Quick and simple.

Now to validate the data is the same and correct.


The Home Server Part 2: Ubuntu & ZFS

So I have my HP Micro Server. It has a modded BIOS. I have installed a spare SSD that I had into the OOD slot. Next step is Ubuntu.

Unfortunately I found the Ubuntu Server 13.10 doesn’t like to work with either of my USB keyboards which makes installing it impossible. So I got a Ubuntu 13.04 ISO and installed it via USB stick to the SSD, then upgraded to 13.10.

Now for ZFS I did the following

sudo -i
apt-get install python-software-properties software-properties-common
add-apt-repository ppa:zfs-native/stable
apt-get update
apt-get install ubuntu-zfs
apt-get install zfs-initramfs
I then put in my 3 500gb test HDD’s that I am going to use as practice for setting up and recovering zfs.
Next I edited /etc/default/zfs to enable automatic mounting/umounting of zfs shares on startup and shutdown, as well as the zfs sharing properties. (Change the top 4 options in the file from no to yes)
Execute the following to get the disk id’s of the disks  you want to use in your zfs pool:
ls -l /dev/disk/by_id
Mine were:
Then to create my RAIDZ pool (the Zfs equivalent of raid5) with my 3x500GB disks:
zpool create -f storage raidz /dev/disk/by-id/ata-WDC_WD5000AAKS-00TMA0_WD-WCAPW1902486 /dev/disk/by-id/ata-WDC_WD5000AAKS-00TMA0_WD-WCAPW1908191 /dev/disk/by-id/ata-SAMSUNG_HD501LJ_S0MUJ13P651726
Then to check its all created
$> zpool list
storage  1.36T   220K  1.36T     0%  1.00x  ONLINE  –
$> zpool status
Now when I ran both of those command the size it gives you is the raw size of the pool which in this case is 1.36TB.
Note also DEDUP is on. When I properly set this up with the final drives I plan to use I will be turning it off on some of my sub file systems.
If I run the following:
$> zfs list
storage   137K   913G  38.6K  /storage
I only see 913GB because of the 1/3 of the data taken to use as a parity.
Finally I am going to create a single zfs filesystem within my zfs filesystem
$> zfs create storage/test
$> zfs list
storage        188K   913G  40.0K  /storage
storage/test  38.6K   913G  38.6K  /storage/test
Next. Creating some data, losing a drive, and rebuilding

The Home Server Part 1: Modding the BIOS

So I got my HP N54L Micro server today, and it is quite simply brilliant, the only flaw with it seems to be a slightly noisy PSU fan, but this will be fine once its in my TV Cabinet.

So the first order of business was to install the modded BIOS to fully enable the extra SATA ports.

Now this would have been quite easy to just download someone else’s modded BIOS and simply apply it using the USB BIOS upgrading key created by the HP Utility

But instead I decided to mod my own BIOS to learn a bit more about them. So my first port of call is to get the Original BIOS (see HP’s website) and a modded BIOS (search the internet for it). Then you need to get the Bios Modding Tool.

Now we have all those tools, we need to decide what we need to do, we need to enable some extra functionality that is currently hidden in the BIOS. Namely we want access to the Southbridge Chipset configuration.

Which we can see is un-ticked. So all we need to do is to tick it, and then save it. (I ticked everything in the end)

Once we are done, we just need to copy it to our USB BIOS updating flash drive, and update the BIOS on the server.

I would recommend having a look around the BIOS with that tool, its really quite interesting.

The Home Server

Last year I decided not to renew my TV License for a myriad of reasons. This has lead me to using Netflix and my considerable DVD and TV/Movie collection a lot more.

I also want to share this content with my other devices remotely, and reliably. Basically what I want is a Netflix but for my collection! My NAS (Synology DS411j) is not powerful enough at the moment to do this, even before we consider transcoding content, and the more advance backup scenarios I would like.

So I then thought that using a AMD A4 CPU, I could build my own HTPC/NAS which could handle this for me. This was 1, looking rather expensive, both in terms of parts (a case which supported 4HDD’s is at least £100) and 2, in power usage.

It was then suggested that I could use a HP N54L micro server which would be able to handle (a colleague with the N36L/N40L could do it, so a N54L should be able to, too) the transcoding, needed for Plex Media Server. Its also significantly cheaper than building something myself, and its power usage is also wonderfully low.

A HP N54L micro server can be got for around £199, you can then get £100 cashback on top of that. So just £99! which is cheaper than a good case (and it comes with a pretty good case).

On top of that, I am adding 16GB of RAM (£115) So a total of £214. Hopefully I can get £100 for my old NAS which means I my outlay is only about £115, which isn’t bad really!

The boot drive will be either the 250GB HDD that comes with the micro server OR a spare 128GB SSD that I have, but I have not decided yet.

I’m going to write a little series, of blog posts as I set it up, the various tools and servers I use, with the custom scripts and bits I plan to use. I will be testing various failure scenarios and how to recover from them (mainly losing a disk, and replacing it). Hopefully by putting them online, they will help someone else, but also help me a few years down the road when I may need it.

So here’s the unspecific list of things I hope to blog about.

  1. Modding the N54L’s BIOS to enable the extra SATA Ports
  2. Installing Ubuntu Server 13.10 & Setting up ZFS
  3. Recovering ZFS from a Drive Failure (simulated by pulling a drive out and wiping it)
  4. SMB Setup
  5. RSync Backup from Clients & Snapshotting
  6. Snapshotting & Clean up
  7. Anything else that comes up long the way



Goodbye Google Apps, Hello (Custom Domains)

I have finally finished moving from Google Apps to Custom Domains (as detailed here). (Well actually the old Google Apps is still running for a few legacy reasons, and forwarding emails)

So I used to use Google Apps Free (now not available) as it was pretty good, and better than Hotmail at the time, plus it was free, and well featured. But Google has been cutting features out of the free version, things which are essential like EAS support (there isn’t a suitable mobile optimised protocol to replace this in my opinion which I will discuss shortly).

So recently the BYOD Wi-Fi at work stopped working for Windows Phone devices as its not officially supported. Since then I have noticed that the Mail applications data usage has been very high (using 250mb+ in 2/3 weeks), which has pushed me over my data limit. This confused me because it never used to do that before we could use the BYOD Wi-Fi at work (which was turned on in February).

Turns out that between then and now, I have installed GDR2 which switches all Google Accounts from EAS to IMAP due to Google cutting EAS support, and IMAP is horrifically inefficient it seems. And its not just my data usage it was eating up either, it was also my battery life. Since moving back to a EAS account, my battery life has shot up.

This is just another example of Google cutting useful features, or making changes to existing products to try and encourage uptake of their other failing products. So in this case it is to ensure people use their Gmail app (on supported platforms which are just Android / iOS at the moment as far as I know), which gives Google more control.

And for those not on Android / iOS they either have to suffer, or switch. If they suffer it is likely they will blame Windows Phone for the high data usage, poor battery life, not Google, which also helps Google get people to switch to Android.