Category Archives: Storage

I recently had one of my freeNAS9.3 report the following issues:

• CRITICAL: Device: /dev/ada2, 8 Currently unreadable (pending) sectors
• CRITICAL: Device: /dev/ada2, 8 Offline uncorrectable sectors

After a bit of reading I decided it best I replace the drive, as I don’t want to take chances since I’m only running RAID-Z1 (for the space), instead of the preferred and safer RAID-Z2.

Then I hit a problem: What physical drive is ada2?  The Gen8 HP Microserver G2020T doesn’t have drive lights to indicate which ones are active.

What I did was took a bit of a pun that the drives are numbered left to right, and it proved correct.

What you really need to do before doing anything, is take a screenshot of the different drive serial #’s. Don’t rely on the drive numbers!!! The reason I say that is because once you pull out a drive, the drive numbers get remapped! When I pulled out ada2, what was previously ada3 became the new ada2! This can get confusing and cause you to pull the wrong drive and screw your data – so concentrate on the serial numbers.

I don’t need to repeat the full instructions, but will link you to them here: http://doc.freenas.org/9.3/freenas_storage.html#replacing-a-failed-drive

Here’s my screenshot of my drives. Note the serial of ada2.

I now offlined ada2 and shutdown FreeNAS in order to pull the drive out, as the drives in the Gen8 HP Microserver are apparently not hot-swappable (that’s something they really should address!). I booted up to make sure that the correct expected drive SERIAL NUMBER disappeared. Notice how what was ada3 is now ada2.

Happy that I have pulled the right drive, I now proceed to shut down again, and then insert the new 4TB Seagate NAS drive. After booting up again:

OK so far so good. I now highlighted the new ada2 as per the screenshot, and then clicked on “Replace”. I then had to confirm that I wanted to replace the original ada2, but I didn’t get a screenshot of that. It then went about reslivering, which is the process of recreating the data on the redundant drive.

For me it got up to about 5% after 10 mins, so figured it would take somewhere between 3 and 4 hours, so kicked it off before I went to bed. The interesting thing is that the old drive’s volume ID (8482932750830730262) is still listed in the array during the resilvering process. It’s as if you can cancel the resilvering and go back to the original drive if you so wished (if you had a failed resilver perhaps?) but I didn’t test this theory. Once resilvering is complete, this old drive/volume reference goes away.

Hope this helps. Happy and safe NASing!

Update: 20 August 2015.  There was an a file in my /tmp directory called “.smartalert” which seemed to contain the source of the alert. I deleted that file and rebooted, and the alarms cleared.

Well I’ve been spending some time with the new HP Microserver Gen8 (G2020T), and wanted to post a few little tips and tricks for using it with FreeNAS

1. Make the internal USB slot external
2. Use MicroSD for swap space
3. Bind CIFS to certain interfaces

Make the Internal USB slot External

With my older HP N40L NAS I was used to accessing the bootable internal USB by opening the front door. With the new NAS, you can’t get to the internal bootable USB without taking the case cover off. I had a spare USB extension cable so I put it to work as per the following photos:

I can now sit it on top of my NAS which gives me the following benefits:

1. I can get to it easy. The rear USB2 are practically blocked by the two NIC ports I’m using, so I don’t have to reach around the back to get to them.
2. It keeps the two front USB ports clear, so I don’t have a stick jutting out, and I can keep them free for if I need to plug in a mouse and keyboard.

Just so you know, you can boot from the external USB ports – the USB2 ones at least. I’m pretty sure I read somewhere that you can only boot from USB2 though – the USB3 ports are made available via OS drivers post-boot. It was quite easy to bend back the bit of metal and feed the plug through.  BTW – you can see my Intel 2-port server NIC there that I use to team two interfaces into my ESXi box for iSCSI disks straight off the NAS.

Use the MicroSD for the FreeBSD/FreeNAS OS Swap Space

This is my favorite tip. See the MicroSD card slot on the photo above? I thought about what use that could be for a dedicated NAS box, where I boot off the USB drive and use the HDD array for storage – surely it’s useless? Well I found a use for it that I’m quite happy with.

What I didn’t know is that by default, not all of your HDD space is used for data storage. In the default settings of FreeNAS (8 and now 9.1) is that it reserves 2GB per-disk for OS swap space.

Where it’s set is SETTINGS -> ADVANCED… “Swap size on each drive in GiB, affects new disks only. Setting this to 0 disables swap creation completely (STRONGLY DISCOURAGED).”

Here is a discussion on the matter.  If a disk fails and you’re using the swap space on it, that could leave your server inoperable. I don’t like that sound of that. What you can do is use the MicroSD for OS swap. This will then leave your HDD’s purely for ZFS data storage.

There are new MicroSD cards from Sandisk  (the creators of the format) that can transfer up to 80MB/s that should be adequate for emergency swap space. Mind you I say “emergency” swap. I am using my box as a NAS with 8GB and soon to be 16GB RAM – I don’t expect to be using swap at all, but like to have it for any emergency needs, which is preferable to the server crashing.  If you are smashing the swap space, then you’d need more RAM, and if you’re still smashing it, then perhaps get a dedicated SSD that can sit in the thin optical drive bay (there are third-party solutions for this).

To set this up this is what you do:-

Set swap size on each drive to “0” before you create any ZFS volumes! Otherwise you’ll have to blow those away and start again.

Put the MicroSD card in. I’m not sure if it’s hot swap, but since it’s a one-off task I do it cold. You can can either check the “dmesg” or via the GUI check that it shows up as a drive. On FreeNAS 9.1 it shows up on my server as “da1”.

On the CLI you type:

swapon /dev/da1

Check that it’s using it:

freenas# swapctl -l
Device:       1024-blocks     Used:
/dev/da1         992000         0

That’s a tiny 1GB MicroSD I pulled from a mobile phone as I didn’t have any others spare. I’ll put the order in for a 16GB Sandisk Extreme soon enough. I think it’s good form to be at least as big as the size of your RAM.
You also need to make sure it enables the swap on boot, so in the FreeNAS web GUI in “System” then under the pre-init scripts section, add a new pre-init script of:

swapon /dev/da1

If your ZFS volumes already have swap on them you might see something like this:

 freenas# swapctl -l
 Device:       1024-blocks     Used:
 /dev/ada0p1.eli   2097152         0
 /dev/ada1p1.eli   2097152         0
 /dev/ada2p1.eli   2097152         0

…because /etc/fstab will have these entries:

 freenas# cat /etc/fstab
 /dev/ufs/FreeNASs1a / ufs ro 1 1
 /dev/ufs/FreeNASs3 /cfg ufs rw,noauto 2 2
 /dev/ufs/FreeNASs4 /data ufs rw,noatime 2 2
 /dev/ada0p1.eli none                    swap            sw              0       0
 /dev/ada1p1.eli none                    swap            sw              0       0
 /dev/ada2p1.eli none                    swap            sw              0       0

You’ll need to build your volumes from scratch to get rid of that swap, unless you know how to hack this and extend the ZFS volume(s) to fill that extra space, which sounds problematic for live data. For me I copied the data off and rebuilt the volumes and datasets, just to be safe.

Bind CIFS to Only Certain Interfaces

With the extra NIC that I didn’t have before, I decided to use the extra port for “Out of Band Management”. This is handy for security so that only computers on the OOBM network can access the web GUI (if you pin the web GUI to a certain IP address). It’s good practice to have all your network devices managed this way where possible. I like to use the default VLAN for this, with general data on VLAN10 and voice traffic on VLAN20, and if I have a dedicated “SAN” (with jumbo frames usually), then I use VLAN30 for this.
Of course I could have set this up previously with VLAN interfaces which is fine from a security perspective, but I do like the idea of OOBM being on a totally separate physical interface for a pure OOBM setup.

You can either bind Samba to individual interfaces, or a range of interfaces by specifying a network. The latter is my preference as it requires less thought. The commands I  use are:

interfaces = 10.10.10.0/24
bind interfaces only = yes

Where you apply this is under “Services –> CIFS –> Auxiliary parameters”

That subnet listed is what I have on my data VLAN10. It’s good for me as my PC has NICs on both Data and OOBM, and means I don’t accidentally mount Samba shares across the OOBM network. BTW that “interfaces” statement doesn’t have to match the subnet mask of your interface. You can just have written “10.0.0.0/8” which depending on your setup might match one or more interfaces. Otherwise you can specify interface names like “bge0”, or “em0” – i.e. the OS-given device name – not the name you gave it with the FreeNAS web GUI.

Well the long-running saga of not having any working USB3.0 on my microserver NAS continues. In my previous post I said “It works perfectly”. Well when I dug a bit further I found that to not be the case. I assumed quite incorrectly that since the 4TB Seagate Expansion Desktop spun up, that it was working at USB3.0 speeds. Bad assumption.

To recap, I needed an external disk drive to my NAS so I can backup files from the NAS using ZFS snapshots that I can browse through with Windows VSS.

Here’s what happens when I plug into Renesas D720201-powered USB3 card:

 umass1: <vendor 0x0bc2 product 0x3312, class 0/0, rev 3.00/3.19, addr 1> on usbus0
 (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
 (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
 (probe0:umass-sim1:1:0:0): Retrying command
 (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
 (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
 (probe0:umass-sim1:1:0:0): Retrying command
 (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
 (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
 (probe0:umass-sim1:1:0:0): Retrying command
 (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
 (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
 (probe0:umass-sim1:1:0:0): Retrying command
 (probe0:umass-sim1:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
 (probe0:umass-sim1:1:0:0): CAM status: CCB request completed with an error
 (probe0:umass-sim1:1:0:0): Error 5, Retries exhausted
 ugen0.2: <vendor 0x0bc2> at usbus0 (disconnected)
 umass1: at uhub0, port 1, addr 1 (disconnected)

I also tried another card that has a newer Renesas D720202 chip – same deal. Here’s what happens plugging into USB2

 ugen2.2: <Seagate> at usbus2
 umass1: <Seagate Expansion Desk, class 0/0, rev 2.10/3.19, addr 2> on usbus2
 da1 at umass-sim1 bus 1 scbus7 target 0 lun 0
 da1: <Seagate Expansion Desk 0319> Fixed Direct Access SCSI-6 device
 da1: 40.000MB/s transfers
 da1: 3815447MB (976754645 4096 byte sectors: 255H 63S/T 60800C)
 da1: quirks=0x2<NO_6_BYTE>

The disk shows up fine, but of course is slow being USB2.

Not happy with that, I bit the bullet and bought the new HP G2020T Gen8 Microserver.  It was a big outlay but when an issue gets under my skin like this – I go all out! I was hoping that somehow it was the Renesas line of chipsets at fault not being properly supported in FreeBSD/FreeNAS. Surely the Intel native chipset will work?!

As an aside, when I was swapping disks over from my old NAS I dropped a disk and it was a write off. I ended up buying four new 4TB Seagate NAS drives as I’m a big fan of Seagate’s reliability.

Unfortunately, even with the newly-released FreeNAS 9.1, I still can’t get USB3.0 working with my Seagate external drive, even on the new Gen8 Microserver! I believe it’s an issue with the FreeBSD XHCI USB3.0 driver. I’ll have to run it at USB2.0 speeds for the time being and keep an eye on the errata for new versions of FreeNAS.

Update: in my new Gen8 Microserver with FreeNAS, I have a MicroSSD card and it shows up as “da1”, and the external USB2 drive shows up as “da2”. In the old NAS, the external USB2 drive shows up as “da1”. Be mindful of this if upgrading NAS and copying your settings across.

I use a HP Microserver N40L for my home NAS, which  run the fantastic FreeNAS operating system and 4x 2TB drives in a RAID-Z2 configuration. I wanted to back up the NAS onto an external drive, with the great feature that is ZFS snapshots.

I started looking at 4TB external drives. The N40L has eSATA and only USB2.0, so I thought that eSATA was my only real option. I quickly learned that eSATA is fast becoming a dead technology, as USB3.0 takes over.

The pure eSATA drives seemed to top out at about 1 or 2TB. The ones that go up to 4TB – the cheapest ones are pure USB3.0. The ones (from Lacie and others) that have eSATA, USB3.0 and FireWire are very expensive, and it is actually cheaper to get a USB3.0-only 4TB drive plus cheap USB3.0 controller card, than it is to get one that supports eSATA. My decision was further solidified by the fact that the new “Gen8” HP Microservers don’t even have eSATA. Buying a USB3.0 controller card became a no-brainer.

I went for a Astrotek 2 Port External / 1 Internal PCI Express USB3.0 Card from the fine folk at Scorpion Technology. The specs as copied from Scorpiontech:

Chipset : RENESAS 720201
PCI Express Base Specification 2.0
2 x USB 3.0 Downstream Port
1 x ICC USB 3.0 (2 x USB 3.0) Internal Downstream Port
1 x ATX Molex Power connector
Compliant with USB specification 3.0 rev. 1.0 compliant
Max. 5 Gb/s transfer rate per port
Supports Super-speed (5 Gb/s), High-speed (480 Mb/s), Full-speed (12 Mb/s), & Low-speed (1.5 Mb/s)
Supports USB 3.0, USB 2.0 & USB 1.1 devices
Compliant with Intel xHCI (eXtensible Host Controller Interface) specifications Revision 1.0
Supports USB Battery Charging Specification Rev. 1.2
Supports PCI Bus Power Management Interface Specification rev. 1.2
Supports USB Wake-up function(total max. 5V/600mA)
1.5A MOSFET Power Switch Protection & Over-current sensing for each port
Supports Hot Swap, Plug & Play
Drivers support Windows XP, Vista, 7, & Server 2008 Release 2 (x86 & x64)
Linux Kernel 2.6.31 native support
Low Profile support

I’ve bolded the items that appeal to me the most.

The Renesas 720201 seems to be one of the best performing USB3.0 controllers, according to a VR Zone review.

Since FreeNAS is built on FreeBSD, any spec sheet that lists an open source OS is good news. I was thus hopeful it would work in FreeNAS.

I plugged in with FreeNAS 8.3 and got no joy. I saw that the card was recognized by the BIOS and was given an IRQ, but it was dead in FreeNAS.  I downloaded a copy of the new FreeNAS 9.1.0RC2 and to my pleasant surprise found that it worked perfectly. Since I’m not ready to upgrade to beta code for my NAS, I’m in for a bit of a wait until I can use my external USB3.0 drive, but shouldn’t take too long.