Emulation Be Gone!
Well I had enough of trying to update the CrashPlan plugin and getting it working with CrashPlan 4.7. I even tried installing a standard jail and using the FreshPorts to get it installed. I got close – I had CrashPlan downloaded but the make script wanted “jdk-8u92-linux-i586.tar.gz” even though that has known vulnerabilities and it wouldn’t take the latest version “jdk-8u102-linux-i586.tar.gz”. I forced the old Java version (security issues and all) and got everything built but then ran into problems with kernel modules, which given a month of Sundays I might have resolved, but I only want to devote half a Sunday to this, so backed out of that when I realised that I was at that cul-de-sac dead end of frustration I’m sure that you know all too well.
Beehyve to the Rescue
So I backed out of the CrashPlan plugin Jail; gave up on the Crashplan standard Jail, and have gone in a whole new direction with FreeBSD’s “Beehyve” which is accessible under FreeNAS 9.10. This is a hypervisor which has kernel support, so you could say it’s type 1, but probably emulates a lot of stuff, so just how “type 1” is for others to say. In any case, it seems to perform flawlessly for me so I’m a happy camper.
Emulating Linux ABI on FreeBSD to me always felt like shoving a square peg through a round hole. Why emulate parts of Linux to run CrashPlan, when you can instead virtualize a whole Linux instance, and run CrashPlan native? This will surely keep compatibility problems to a minimum. Not only that, but I can move my TVHeadend to it as well, and anything else that I need to run on a Linux server. I can’t see myself ever going back to the plugin setup for CrashPlan.
Setting up the Beehyve Environment
It was reasonably straightforward setting up Ubuntu Server 16.04 (“Xenial Xerus”), which I will use to host my CrashPlan server, and also my TVHeadend server, and anything else that I really need Ubuntu for.
Speedy Alias – “iohyve” becomes “io”
You can configure beehive directly, but you’re far better served by using the “iohyve” scripts. Now here’s the thing – I hate typing. I am also a clumsy typer with bent fingers, and find “iohyve” particularly annoying to type. You can do what I do and alias “iohyve” to “io” to make things easier. Send a “which io” to make sure that the alias isn’t used in your path already, and then add it to your “~/.bashrc” if you’re using bash:
sarlacc# which io #make sure that 'io' isn't used for any other commands sarlacc# cat ~/.bashrc | grep iohyve alias io='iohyve' # the alias I added to ~/.bashrc
After adding the alias, log out and log back in, or just source the rc file: . ~/.bashrc
All my subsequent “iohyve” commands will just show “io”.
Initial Parameters
Beehyve needs to know 3 things:
- Where to store its files?
- Which NIC to bridge to?
- If it should start up the kernel modules? (yes… yes it should!)
Configure the answer to those three questions with the following:
io setup pool=<ZFS pool> kmod=1 net=<bridged NIC> #kmod=1 means yes, 0 means no. e.g. io setup pool=volume1 kmod=1 net=vlan10 Setting up iohyve pool... On FreeNAS installation. Checking for symbolic link to /iohyve from /mnt/iohyve... Symbolic link to /iohyve from /mnt/iohyve successfully created. Loading kernel modules... bridge0 is already enabled on this machine... Setting up correct sysctl value... net.link.tap.up_on_open: 0 -> 1
Some older docs say that on FreeNAS you need to ln -s /mnt/iohyve /iohyve but as you can see above that’s already added. If you add the symlink manually it’ll create a weird circular sym linking.
Files and Folders
Run this to see that the folder structure is setup:
sarlacc# zfs list | grep iohyve volume1/iohyve 21.4G 2.46T 140K /mnt/iohyve volume1/iohyve/Firmware 140K 2.46T 140K /mnt/iohyve/Firmware volume1/iohyve/ISO 771M 2.46T 151K /mnt/iohyve/ISO volume1/iohyve/ISO/FreeBSD-10.3-RELEASE-amd64-bootonly.iso 116M 2.46T 116M /mnt/iohyve/ISO/FreeBSD-10.3-RELEASE-amd64-bootonly.iso volume1/iohyve/ISO/ubuntu-16.04.1-server-amd64.iso 655M 2.46T 655M /mnt/iohyve/ISO/ubuntu-16.04.1-server-amd64.iso volume1/iohyve/ubusrv16 20.6G 2.46T 140K /mnt/iohyve/ubusrv16 volume1/iohyve/ubusrv16/disk0 20.6G 2.48T 2.66G -
You should just have the first three paths – the rest is stuff I’ve setup later on in this guide.
The Kernel Modules
You can check that the kernel modules are loaded with this:
sarlacc# kldstat Id Refs Address Size Name 1 94 0xffffffff80200000 18b4000 kernel 2 1 0xffffffff81d9f000 ffd8c ispfw.ko 3 1 0xffffffff82021000 f947 geom_mirror.ko 4 1 0xffffffff82031000 46a1 geom_stripe.ko 5 1 0xffffffff82036000 ffca geom_raid3.ko 6 1 0xffffffff82046000 ec6a geom_raid5.ko 7 1 0xffffffff82055000 574f geom_gate.ko 8 1 0xffffffff8205b000 4a33 geom_multipath.ko 9 1 0xffffffff82060000 5718 fdescfs.ko 10 1 0xffffffff82066000 89d dtraceall.ko 11 10 0xffffffff82067000 3ad67 dtrace.ko 12 1 0xffffffff820a2000 4638 dtmalloc.ko 13 1 0xffffffff820a7000 225b dtnfscl.ko 14 1 0xffffffff820aa000 63d7 fbt.ko 15 1 0xffffffff820b1000 579a4 fasttrap.ko 16 1 0xffffffff82109000 49cb lockstat.ko 17 1 0xffffffff8210e000 162f sdt.ko 18 1 0xffffffff82110000 d8d8 systrace.ko 19 1 0xffffffff8211e000 d494 systrace_freebsd32.ko 20 1 0xffffffff8212c000 4da3 profile.ko 21 1 0xffffffff82131000 7fdf ipmi.ko 22 1 0xffffffff82139000 b3c smbus.ko 23 1 0xffffffff8213a000 1a62a hwpmc.ko 24 1 0xffffffff82155000 2b80 uhid.ko 25 2 0xffffffff82158000 2b32 vboxnetflt.ko 26 2 0xffffffff8215b000 45320 vboxdrv.ko 27 1 0xffffffff821a1000 41ca ng_ether.ko 28 1 0xffffffff821a6000 3fd4 vboxnetadp.ko 29 1 0xffffffff821aa000 3567 ums.ko 30 1 0xffffffff821ae000 a684 linprocfs.ko 31 1 0xffffffff821b9000 670b linux_common.ko 32 1 0xffffffff821c0000 1b140b vmm.ko 33 1 0xffffffff82372000 2ebb nmdm.ko 34 1 0xffffffff82375000 1fe1 daemon_saver.ko
If vmm.ko and nmdm are there, you’re golden.
MTU – Danger Will Robinson!
Now the “bridged NIC” is the physical or logical NIC that carries the IP address of the network that you want your virtual machine to bridge to – not the bridged interface. For my home setup I share a VLAN10 (data) and a VLAN99 (management) on a single physical interface – bge0. Why do I do this? Well my switches and routers only have management IPs on VLAN99, and my computer is the only one on VLAN99, so that’s added security. Plus I do it, because I am a network engineer, and because I can 🙂
Now when you have VLAN interfaces you can run into MTU problems, unless you up the MTU to account for the extra 4 bytes of VLAN tag overhead. In FreeNAS GUI, I set “mtu 1504” on any interface I run VLANs on, so that the VLANs can get 1500 bytes MTU.
The automatically created bridge0 interface interits this MTU:
sarlacc# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1504
description: iohyve-bridge
ether 02:f3:f6:80:91:00
nd6 options=1<PERFORMNUD>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000000
member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 20000
sarlacc# ifconfig tap0
tap0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1504
description: iohyve-ubusrv16
options=80000<LINKSTATE>
ether 00:bd:1b:3e:01:00
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
Opened by PID 3694
That tap0 is originally created by iohyve as 1500 bytes, and fails to add to the bridge0 because of the MTU mismatch. In order to get it into the bridge0, I had to do this:
ifconfig tap0 mtu 1504 ifconfig tap0 promisc # not sure if this was necessary but added anyway ifconfig bridge0 addm tap0
Surviving Reboots
You want these settings to survive reboots, so add these in the GUI to your “System” > “Tunables”.
Unfortunately I haven’t worked out how to do the tap0 MTU fix just yet, so I’m manually doing that at reboot just for now. I’d like this to be “fixed” by iohyve, but if all else fails I could add a pre or post init script that just runs the commands that way.
Installing Ubuntu 16.04 “Xenial Xerus”
Either FTP fetch the install media or add the path:
io fetch ftp://ftp.iinet.net.au/pub/ubuntu-releases/16.04.1/ubuntu-16.04.1-server-amd64.iso io cpiso /mnt/volume1/files/software/ISOs/Ubuntu/ubuntu-16.04.1-server-amd64.iso
Once downloaded or copied see that it’s listed:
sarlacc# io isolist Listing ISO's... FreeBSD-10.3-RELEASE-amd64-bootonly.iso ubuntu-16.04.1-server-amd64.iso
Now create the VM and set its parameters (I call my VM ubusrv16 for Ubuntu Server 16.x):
sarlacc#io create ubusrv16 20G sarlacc#io set ubusrv16 loader=grub-bhyve os=d8lvm ram=2G cpu=1 con=nmdm1 sarlacc#io list Guest VMM? Running rcboot? Description ubusrv16 NO NO NO Sun Jul 24 11:04:26 AEST 2016
Use “os=debian” if not using LVM. If using LVM, use “os=d8lvm”
I just give it one CPU, and 2Gigs of RAM. The console will be nmdm1 if it’s the first VM.
Do the install, and use another SSH session to attach to the console:
io install ubusrv16 ubuntu-16.04.1-server-amd64.iso io console ubusrv16 #handy to do this in another window
Configuring VM to Start at Reboot
One criticism I’ve heard of VirtualBox is that you can’t start the VMs on reboot. I haven’t verified this though. The good thing with Beehyve is that you can start a VM on reboot:
io set ubusrv16 boot=1
🙂
Let me know you you go in the comments.