My blog, this blog, never really took off. It was a great idea but I think I was even too much of a geek for my wife to make sense of.
Recently I started a new home network project that I have decided to write a little about, so I might as well publish my ramblings here since there's really no where else...
Geek said.
Pretty sure She said, "whatever".
I love her. She humors me.
Monday, April 8, 2013
The Great Virtual Sandbox Build of 2013
For many years when I was first getting my feet wet in IT I kept multiple old computer systems and referred to them as my digital sandbox. Much like the original sandbox I played in as a child only much cleaner if you ignore the dust bunnies…oh who am I kidding, I played more with LEGO than sand but the concept is the same. My sandbox allowed me to build, tear down, rebuild and repeat. The possibilities were limitless.
Now that I have been in the field for 15 years and have started raising a family the play time is also somewhat limited, but even more so storage is extremely limited. I still have boxes full of computer parts that I need to find a new home for, but gone are the days of keeping 3 old computers just for the heck of it. However, my desire to build and test proof of concepts remain, so what is a geek to do?
Enter my latest adventure, "The Great Virtual Sandbox Build of 2013".
Computer system hardware performance has for the most part out paced the requirements of software and advancements in virtualization have been amazing. This led me to the idea that the computer I have running in my basement 24/7/365 and call my "file server" could be...so…much…more. I recently migrated a Cisco Unified Communications Cluster to Cisco UCS hardware which involved virtualizing multiple UCM application servers on ESX.
*insert geek brain gear noises*
VMWare has a version of ESXi that is free, albeit feature stripped, but certainly capable of providing the platform I was looking for. Support for a single CPU with any number of cores and up to 32GB of memory? That should do. BUT, what about hardware compatibility? Oh crap… Well let's just figure out what I want to be able to do with this thing.
1. I need to maintain my file server which is running unRAID and had been doing so for nearly 300 days without a reboot at the time I took it down to complete this build.
- To accomplish this unRAID needs direct disk access using VMWare's Direct I/O which relies on the virtualization features of the CPU and the motherboard. Eeek! Finding the right combination took a significant amount of research and reading.
2. I have a vibrant Apple culture in my house with multiple iDevices and a couple of MacBook Pros so naturally I will want to run a Mac Server and play with things like Profile Manager.
- Apple only licenses Mac OS to run on Apple hardware. Sigh. That won't work well with unRAID. Doh! Well then I guess I may have to avoid reading the EULA and seek another solution.
- Since ESXi natively supports Mac OS it would be just a matter of unlocking that functionality on non-Apple hardware.
- Don't forget Mac OS only natively supports Intel processors and chipsets. Bummer. I normally use AMD CPUs to cut costs, but in this case compatibility rules. Patching a Hackintosh VM is not on my bucket list.
3. I have a long sorted history with Windows Server and Active Directory and as much as I could care less about it I have to work with it regularly and deal with its existence. Since it is so widely used I need to test integrations with other systems.
- Windows server licenses are expensive! Cracking it may or may not be possible, but I really hate being a pirate. Except on Halloween, then it's my purpose. Argh Matey! Trick or Beer! Thankfully I have a .EDU email and acquired a legitimate license through Dreamspark. Wait, that's for students? Shame. On. Me. At least I'm using it for educational purposes, right?
4. I love Linux servers. I love applications that run on Linux. I pretty much love everything about Linux. So obviously I'll be running a Linux server VM, that's a no brainer.
- FREE FREE FREE WOO HOO HOO!!! 'Nuf said.
5. I begrudgingly admit that I am a VoIP admin. Oh wait, I already did that. That's what got me into this thought process in the first place. I would like to test CUCM in various scenarios and not break my production environment that has 9 or so sites with 1200+ phones….oh and that's just my part-time job…
- Loading CUCM in a VM on non-Cisco hardware isn't overly complicated but has some small issues. Best part is it installs with a DEMO license that supports like 6 phones. More than what I need.
Perfect. That's what I "need" to be able to do with my virtual sandbox. The hardware must also support what I want to be able to do now and in the future.
- Processor: Intel. Fast, low power CPU with at least 4 cores, VT-d support (that direct hardware access) and preferably hyperthreading.
- Motherboard: Lots of SATA connectors, PCI-E and PCI slots, at least 32GB memory, but most importantly VT-d support. I can't stress the importance of this project relying on direct hardware access.
- Memory: Minimum 16GB, the fastest supported natively by the motherboard. You can have your over-clocking, I'll keep my stability. My 292 days runtime in unRAID was important to me.
- Power supply: High wattage, 80plus gold certified, this thing runs 24/7/365 it needs to be efficient.
- SATA controller: I'm going to need a minimum of 9 SATA ports to support the existing hard drives (1 ESXi datastore, 8 unRAID) going into this system.
That pretty much covers the main points of the hardware which guided me to a build sheet. After literally months of research and some waiting to see if others have tried the hardware (specifically the motherboard) I settled on the following parts:
- ASRock Z77 Extreme4
- Intel Core i7-3770S
- G.Skill RipJaws X Series 2x8GB -- At some point 2 more will be added.
- Rosewill CAPSTONE Series CAPSTONE-750-M 750W
- ADATA Value-Driven S102 Pro 16GB -- ESXi Boot Drive
- IBM M1015 -- Crossflashed to LSI 9211-8i, 2x4 port SAS/SATA controller
That pretty much sizes it up since I was re-using existing hard drives and computer case.
First, the ASRock Z77 Extreme4, what I learned:
- The onboard NIC requires a custom ESXi installer or it just doesn't work. It'll show up as there, but it won't pass any data.
- Only 6 of the 8 SATA ports passthrough properly to ESXi virtual machines in the default EFI configuration. Since I was putting in the IBM M1015 this didn't matter so I didn't try to figure out why the other 2 didn't.
- I used a USB3 header cable to put the ADATA flash drive inside the case rather than have it dangle out the back and get snapped off accidentally.
Second, well all I have left is the experience, the rest of the hardware worked without quirks.
I am not extremely knowledgeable with ESXi, that was part of the reason for this build so I took the first couple days breaking it. I re-installed ESXi at least 6 times because I made an erroneous change that broke it and rather than take the time to figure out how to fix it I just started from scratch and considered it a lesson learned. I did all this in a temporary case so that my unRAID server could continue to clock up-time.
When I finally settled on the ESXi setup, my SATA controller hadn't arrived. I decided to use a couple other PCI cards I had on hand for the time being. I tore down my unRAID box and replaced the hardware, hooked everything up and I thought I was good to go. I booted up ESXi, everything is happy, passed through the SATA controllers to the unRAID VM and proceeded to boot up the VM. This is where I learned another valuable lesson. Only 4 of my 8 unRAID drives showed up. Crap. Turns out only one of me 2 SATA controllers were supported by unRAID. I should have waited until the IBM card arrived. When it finally did I powered up the VM and none of my drives showed up in unRAID. Double crap! Fast solution, upgrade unRAID from 4.7 to 5.0RC(whatever the latest was). I was planning to wait until version 5 was final, but oh well. I needed to start clocking uptime again. After the unRAID upgrade all my drives showed up and everything came online.
I have the basic virtual machines built and operating and have started building out a "small business" like virtual server farm for my home network. Slowly I am tearing down the old environment and replacing it. I had been using a Raspberry Pi for DHCP/DNS, but that has been replaced by the Windows Server since I now have an Active Directory operating.
My first virtual environmental challenge will be the integration of Active Directory, Open Directory and DNS. Both Windows and Mac server will run their respective directory services as well as DNS and shall permit zone transfers.
Let the good times roll…
Geek said.
Now that I have been in the field for 15 years and have started raising a family the play time is also somewhat limited, but even more so storage is extremely limited. I still have boxes full of computer parts that I need to find a new home for, but gone are the days of keeping 3 old computers just for the heck of it. However, my desire to build and test proof of concepts remain, so what is a geek to do?
Enter my latest adventure, "The Great Virtual Sandbox Build of 2013".
Computer system hardware performance has for the most part out paced the requirements of software and advancements in virtualization have been amazing. This led me to the idea that the computer I have running in my basement 24/7/365 and call my "file server" could be...so…much…more. I recently migrated a Cisco Unified Communications Cluster to Cisco UCS hardware which involved virtualizing multiple UCM application servers on ESX.
*insert geek brain gear noises*
VMWare has a version of ESXi that is free, albeit feature stripped, but certainly capable of providing the platform I was looking for. Support for a single CPU with any number of cores and up to 32GB of memory? That should do. BUT, what about hardware compatibility? Oh crap… Well let's just figure out what I want to be able to do with this thing.
1. I need to maintain my file server which is running unRAID and had been doing so for nearly 300 days without a reboot at the time I took it down to complete this build.
- To accomplish this unRAID needs direct disk access using VMWare's Direct I/O which relies on the virtualization features of the CPU and the motherboard. Eeek! Finding the right combination took a significant amount of research and reading.
2. I have a vibrant Apple culture in my house with multiple iDevices and a couple of MacBook Pros so naturally I will want to run a Mac Server and play with things like Profile Manager.
- Apple only licenses Mac OS to run on Apple hardware. Sigh. That won't work well with unRAID. Doh! Well then I guess I may have to avoid reading the EULA and seek another solution.
- Since ESXi natively supports Mac OS it would be just a matter of unlocking that functionality on non-Apple hardware.
- Don't forget Mac OS only natively supports Intel processors and chipsets. Bummer. I normally use AMD CPUs to cut costs, but in this case compatibility rules. Patching a Hackintosh VM is not on my bucket list.
3. I have a long sorted history with Windows Server and Active Directory and as much as I could care less about it I have to work with it regularly and deal with its existence. Since it is so widely used I need to test integrations with other systems.
- Windows server licenses are expensive! Cracking it may or may not be possible, but I really hate being a pirate. Except on Halloween, then it's my purpose. Argh Matey! Trick or Beer! Thankfully I have a .EDU email and acquired a legitimate license through Dreamspark. Wait, that's for students? Shame. On. Me. At least I'm using it for educational purposes, right?
4. I love Linux servers. I love applications that run on Linux. I pretty much love everything about Linux. So obviously I'll be running a Linux server VM, that's a no brainer.
- FREE FREE FREE WOO HOO HOO!!! 'Nuf said.
5. I begrudgingly admit that I am a VoIP admin. Oh wait, I already did that. That's what got me into this thought process in the first place. I would like to test CUCM in various scenarios and not break my production environment that has 9 or so sites with 1200+ phones….oh and that's just my part-time job…
- Loading CUCM in a VM on non-Cisco hardware isn't overly complicated but has some small issues. Best part is it installs with a DEMO license that supports like 6 phones. More than what I need.
Perfect. That's what I "need" to be able to do with my virtual sandbox. The hardware must also support what I want to be able to do now and in the future.
- Processor: Intel. Fast, low power CPU with at least 4 cores, VT-d support (that direct hardware access) and preferably hyperthreading.
- Motherboard: Lots of SATA connectors, PCI-E and PCI slots, at least 32GB memory, but most importantly VT-d support. I can't stress the importance of this project relying on direct hardware access.
- Memory: Minimum 16GB, the fastest supported natively by the motherboard. You can have your over-clocking, I'll keep my stability. My 292 days runtime in unRAID was important to me.
- Power supply: High wattage, 80plus gold certified, this thing runs 24/7/365 it needs to be efficient.
- SATA controller: I'm going to need a minimum of 9 SATA ports to support the existing hard drives (1 ESXi datastore, 8 unRAID) going into this system.
That pretty much covers the main points of the hardware which guided me to a build sheet. After literally months of research and some waiting to see if others have tried the hardware (specifically the motherboard) I settled on the following parts:
- ASRock Z77 Extreme4
- Intel Core i7-3770S
- G.Skill RipJaws X Series 2x8GB -- At some point 2 more will be added.
- Rosewill CAPSTONE Series CAPSTONE-750-M 750W
- ADATA Value-Driven S102 Pro 16GB -- ESXi Boot Drive
- IBM M1015 -- Crossflashed to LSI 9211-8i, 2x4 port SAS/SATA controller
That pretty much sizes it up since I was re-using existing hard drives and computer case.
First, the ASRock Z77 Extreme4, what I learned:
- The onboard NIC requires a custom ESXi installer or it just doesn't work. It'll show up as there, but it won't pass any data.
- Only 6 of the 8 SATA ports passthrough properly to ESXi virtual machines in the default EFI configuration. Since I was putting in the IBM M1015 this didn't matter so I didn't try to figure out why the other 2 didn't.
- I used a USB3 header cable to put the ADATA flash drive inside the case rather than have it dangle out the back and get snapped off accidentally.
Second, well all I have left is the experience, the rest of the hardware worked without quirks.
I am not extremely knowledgeable with ESXi, that was part of the reason for this build so I took the first couple days breaking it. I re-installed ESXi at least 6 times because I made an erroneous change that broke it and rather than take the time to figure out how to fix it I just started from scratch and considered it a lesson learned. I did all this in a temporary case so that my unRAID server could continue to clock up-time.
When I finally settled on the ESXi setup, my SATA controller hadn't arrived. I decided to use a couple other PCI cards I had on hand for the time being. I tore down my unRAID box and replaced the hardware, hooked everything up and I thought I was good to go. I booted up ESXi, everything is happy, passed through the SATA controllers to the unRAID VM and proceeded to boot up the VM. This is where I learned another valuable lesson. Only 4 of my 8 unRAID drives showed up. Crap. Turns out only one of me 2 SATA controllers were supported by unRAID. I should have waited until the IBM card arrived. When it finally did I powered up the VM and none of my drives showed up in unRAID. Double crap! Fast solution, upgrade unRAID from 4.7 to 5.0RC(whatever the latest was). I was planning to wait until version 5 was final, but oh well. I needed to start clocking uptime again. After the unRAID upgrade all my drives showed up and everything came online.
I have the basic virtual machines built and operating and have started building out a "small business" like virtual server farm for my home network. Slowly I am tearing down the old environment and replacing it. I had been using a Raspberry Pi for DHCP/DNS, but that has been replaced by the Windows Server since I now have an Active Directory operating.
My first virtual environmental challenge will be the integration of Active Directory, Open Directory and DNS. Both Windows and Mac server will run their respective directory services as well as DNS and shall permit zone transfers.
Let the good times roll…
Geek said.
Subscribe to:
Posts (Atom)