Camp Kiwi Data Center: Difference between revisions
No edit summary |
|||
| Line 95: | Line 95: | ||
<hovergallery maxhoverwidth=1200 maxhoverheight=1024> | <hovergallery maxhoverwidth=1200 maxhoverheight=1024> | ||
Image:IMG_6821.jpeg|Carpet after plastic was removed | |||
Image:IMG_6820.jpeg|Carpet after plastic was removed | |||
Image:floor_power.jpg|Power Outlets in Floor under racks | Image:floor_power.jpg|Power Outlets in Floor under racks | ||
Image:flor_squares.jpg|Carpet Design | Image:flor_squares.jpg|Carpet Design | ||
Revision as of 18:52, 17 May 2020
Introduction
I started my "home lab" experience at our home on Wandering Way, by pulling a couple of Ethernet cables from one side of the house, where the cable came into the building, to my office. I also pulled one from there to a WiFi access point on the other side, and a computer in the kitchen island.
From there we moved to Top Flite where I converted a pretty good space (maybe 10'x8'?) into a mini data center. My brother-in-law is in the surplus business and ended up with some old Great Lakes 45U racks he didn't need, so I picked one up. I actually picked up 3, but I only used one over there. It's not ideal, being a "Security and Sound" GL840S2-2436 rack, it doesn't have any cable management features, but I got them basically free (thanks Steve!), so who's arguing?
That Top Flite house was built in the early 2000's, and was wired with Cat5e wire to most of it's rooms, alongside dual RG6 coax, I think for the phone lines. These were homerun back to the utility space that I'd taken over for the data center, so I started getting rack-mount switches, routers, and shelves and mounted my cable modem and mac mini server in there. I upgraded to a USG Pro and USW-16, and eventually a USW-48-750 to power my WiFi access points and some cameras. I became an amateur wire-puller as I added ceiling-mounted UAP-AC-Pros in 4 places around the house and started to monitor the property with Foscam PTZ cameras, and added dual fiber drops to a couple of computer locations.
Eventually I added a portable 1 ton AC unit and QNAP NAS for a Plex server, and I was really a Home Labber! The heat from the AC was ducted to the attached garage, which as luck would have it was on the adjoining wall. Somewhere along the way, Cincinnati Bell started offering 1 Gbps FTTH service (250 Mbps upload) and I snapped that up too. That upload speed was particularly important for my Plex server.
Then it came time to move again. I got my wife to agree that we would only look at houses that had fiber service, and that I would get a budget to install good Ethernet cabling. And so it began...
Stats
The data center features Ubiquiti Unifi "pro-sumer' networking gear, with:
- 96 Cat6a copper network runs, and 2 miles of cable
- 12 OM4 fiber-optic network runs (capable of 40 or 100 gigabit per second)
- 8-20A power circuits
- 1.5 tons of AC cooling (typically set to 65F, which keeps rack outlet temps -top of rack- below 75F), all-weather capable
- Core network is dual 10 gigabit per second fiber optic
- Primary internet service to house 1 gigabit per second fiber optic (Cincinnati Bell Fioptics)
- Backup internet service to house 200 megabit per second cable (Spectrum)
- File server is 120 terabytes of disk space (running Plex and Nextcloud personal cloud service) on QNAP TVS-1271u
- 1U Redundant Virtualization Servers are Dell R620 (local services, DNS, NTP, Ad Blocking, Home Automation)
- 2U Redundant Virtualization Servers are Dell R720 (Cloud Data Server, Movie Streaming, eMail, Web, Wiki)
- 2U Storage Area Network Server is Dell R720XD (6x6TB VM storage, 6x12TB Backup)
- 2 Rijer 8 port VGA KVM Switches with remote (USB) switching on my desk
- 7 Sonos network audio amps for basement level audio
- 4 Sonos network audio amps for main floor audio
- 3 Tivo Bolt DVRs (rack mounted) for entertainment
- 6 Tivo Mini Vox (Gen3) streaming from the Bolts above (with Netflix/Hulu/Plex/YouTube)
- 16 UVC camera security system with on site video backup
- 5 Ring Floodlight cameras
- 3 Ring Doorbell cameras
- 4 Ring Doorbell extender chimes
- 21 Network switches (8 in-rack)
- 7 UniFi HD WiFi Access Points
- 150-175 typically associated network clients
Origin of the Name
There's this guy named Dave. I used to work with this guy, and he is a bit... eccentric. Anyway, he managed a data center at the office, and had some folks supporting it. Some of these folks were not native English speakers, and he had a Monday morning telephone conversation where it got confused as to what they were saying about eating a can of peaches versus visiting "Camp Peaches", presumably like a summer camp for Scouts. In a fit of brilliance he decided to name his data center "Camp Peaches". He even acquired signage to proclaim same. When he took on a new role, I inherited this sign (our IT organization did not see the brilliance in the naming scheme - however, it is still universally known as Camp Peaches).
Camp Peaches, origin of the 'Camp Kiwi' moniker
Finished Pictures
Here are some snaps of the finished product. I'll be adding these as each section comes online. This section is an overview, the details of everything are below in the 'parts and bits' section.
Workstation
The basic Data Center is now online. As you see in the below pictures, I have a small workstation inside the data center. This is not my office, I have a full office elsewhere - and it's too cold and noisy (not as bad as you would imagine, but noisy) in the data center to spend a lot of time there. But, I have a set of KVM switches connected to my QNAP NAS, Mac Mini webserver, the Dell servers, and my UniFi Application server to allow me to interface with these computers. The workstation allows me to not only administer the servers, but also monitor the security cameras from around the house. I also have a monitor connected to a HDMI switcher that allows me to view each of my Tivo Bolt DVRs and validate they are functioning properly on a separate display. Note in the most recent pictures I now have a "stand-up desk" since I don't have much room for a chair, and really don't want to sit in here for long periods.
Monitors for Tivo and server control, standing desk
Security Camera Configuration
Cameras-All View
Cameras Interior
Cameras Exterior
The Design
Dual 8 port KVM switches for server monitoring
Climate Control
Data center temperature is managed with a 1.5 ton Carrier/Trane (40MAQB18B--331/38MAQB18R--301) mini split AC unit. This unit is capable of cooling the data center all winter with outside air temperatures well below freezing (the servers will always need some cool air). The wall-mounted thermostat is on the hot-aisle to check the server exhaust air temperature, and is set to keep that temperature in the 62 degree Fahrenheit range. Also, the temperature is monitored continually with an Sensor Push temperature probe on a shelf in the hot aisle, as well as CyberPower UPS environment monitors (and more Sensor Push units for iOS reporting) on the top of each rack. Both of these device types can SMS/push notify me if the temperature or humidity gets out of range. The server rack has the most extreme delta-T, with exhaust temps at the top of the rack at about 72 F, whereas the Network rack is about 66F, the entertainment rack 68F, and the hot asile at the 65 F set point (cold aisle is typically upper 50 F).
HVAC
Mini split AC compressor
Mini split outside unit mounted above the snow line so it cool data center all winter long
Mini split indoor unit on cold side of data center
Thermostat
Thermostat final
Power
A new subpanel was mounted adjacent to the data center to provide power to the server space and the new AC unit. This panel provides eight 20A circuits to the data center below the raised flooring. 2 circuits are under the networking rack providing primary and backup power, and 3 circuits to each server cabinet. Each rack is provided with a CyberPower 1500VA UPS (OR1500LCDRM1U), with the networking and server racks having both primary and secondary/backup units to feed the dual-power-supply gear in those racks.
Additional power management capability is coming (DONE-July 2019). I've started by installing Eve Energy HomeKit power monitors on the primary and backup legs of each machine so I can track total power usage, and I've noted that the 1500VA CyperPower UPS are over-taxed in their current configurations and provide only a few minutes of run time. I therefore plan to get CyberPower monitored PDUs for each circuit in the rack, and for each UPS. Further, I plan to install one UPS for each Dell server at a minimum, and connect the USB output to that server and run "netuptools" to signal the host to shutdown the VMs and power off when power gets critical. UPDATE: This is now done, in addition to the 900W CyberPower units, I have added four 1,350W CyberPower (2U 1500VA model PR1500LCDRT2U) UPS units in the server rack. Each server has a primary and secondary leg on different UPS units, and each UPS is fed from a different circuit breaker. I also added CyberPower 1U rack mount monitored PDU so I can easily see the amp loading for each circuit. I have not yet purchased the network expansions for these units, but I plan to. However, with this additional power I now have 30 minutes minimum runtime on power failure. I replaced all of the black power cables with color-coded ones to make it easier to quickly see what servers are powered by what UPS (and consequently which circuit). I've also started to play with Proxmox's high-availability settings to migrate running VMs to other servers when a system goes down. Right now each UPS is connected via USB to one server and networkUPStools is running in standalone mode to signal the host to shutdown when the battery goes critical. I will set them up to run in a master/slave configuration later so that the host will only shutdown when both legs of power go critical, I just haven't had the time yet.
Power Outlets in Floor under racks
Subpanel
CyberPower PR1500LCDRT2U
Color-Coded power cables
Power Monitoring
Older UPS picture
Flooring
First, the floor was raised 4" (pressure treated 2"x4" studs on edge) and placed on 12" centers for strength with 3/4" plywood covering. Within this flooring, in addition to power, are trenches with removable covers for routing data cabling from the racks to the patch panel as well as KVM and HDMI wiring to the workstation. The floor is covered with Flor carpet squares which are thin and easy to roll a rack across and durable, but also attractive. The tiles are just under 20" square (50cm x 50cm), and the style we picked was "Lasting Grateness" in indigo and bone, simulating an iron grate.
Carpet after plastic was removed
Carpet after plastic was removed
Power Outlets in Floor under racks
Carpet Design
Wiring trench in floor
Open trench before wire
Trench nearly full
Everything but the fiber optics
Servers and Racks
I have 3 racks for my equipment. The first rack houses all of the primary networking gear, the middle rack my servers, and the last rack has the entertainment hardware like Sonos sounds systems and Tivo networked DVR (digital video recorders for cable and over-the-air TV). Each rack has backup battery power in the event of electrical outage and lightening. The fronts feature RGB LED "mood" lighting, and backs have white LED lighting for task illumination, which is usually off but great for maintanence.
I have a few servers. The most impressive of which is a QNAP 12 bay NAS, or 'network attached storage' (UPDATE-this is now my weakest server! although it is still the primary storage unit). This has PC class CPU and twelve 10 Terabyte hard drives, two of which serve for redundancy in case of failure, so I net out at around 100 TB of total storage. This computer hosts my plex server (UPDATE-it only servers the files now, the Plex server is a VM in my ProxMox cluster) and library as well as a virtualization environment that runs a private cloud (NextCloud) instance. This cloud keeps the family's files and picture, replicating them across our devices, as well as a shared family calendar and contacts/address book (UPDATE-the NextCloud private cloud is also now in my ProxMox cluster). The Plex content is available on our Tivos as well as iOS, Playstation, and other devices in home and away.
The other main server that I host is my mail, web, and Wiki server, ferrellmac.com running on a Mac Mini. Apple is abandoning the Server product, so these will soon move to an Open Source Linux system.
My final current server is a Ubiquiti UniFi Application Server. This is a purpose-built system in the UniFi line that hosts both the UniFi "software defined network" and the UniFi video security camera NVR (network video recorder). This recorder supports our 16 security cameras and all of the network configurations via a "single pane of glass".
UPDATE: With the coming sunset of Apple's server product line, I've move all of my services to Linux (mostly Ubuntu 16.04, although I'm sure I'll need to gradually move to 18.04 soon) virtual machines running in a cluster of 4 ProxMox host machines. ProxMox is a Type 1 Hypervisor with high-availability, so I can live-migrate a server from physical host to another without the guest OS being aware that it was moved. This is great as it allows me to keep my Plex server up even while upgrading and rebooting the host machines. 4 Host machines means that I always have a quorum of live hosts even if a system has to go down. All of the hosts are 12 generating Dell rack mount Xeon server hardware with 192 GB of ram, 10G networking cards, and dual redundant power supplies. I have another Dell (R720XD) running FreeNAS as the SAN storage for VM images, ISOs, and backups.
I plan to get another R720XD and fit it out with twelve 12TB drives so I can eventually move off of my QNAP server. It's just shown itself to be too limited for what I wanted to do. It's virtualization was slow/weak and problematic - I had several instances where VMs were corrupted and struggled to get the backup/snapshots back, and the transfer and Plex encoding speeds were just not up to my needs. The FreeNAS server so far has shown itself to be superior in every way.
"Cold Aisle" or front of rack near mini-split HVAC unit
"Hot Aisle" - Back of racks (white LED lighting provided for service)
Front of racks (multi-colored LED for mood lighting)
Code Red!
Entertainment Rack - Tivo DVRs, Sonos connect amps for house in-ceiling audio
Bottom of Entertainment Rack
Server rack - Running virtual servers for private cloud, web hosting, email, security camera monitoring and recording, etc.
The business end of the Data Center
UPS backup power for servers
Back of servers with color-coded power cables
Bottom of Server rack, before additional servers installed
Networking rack
Bottom of Networking rack
iPhone WiFi performance test 377Mbps down / 239 Mbps up
Wired speed test - 940Mbps down, 240Mbps up
File copy speed to QNAP NAS, 237.1MB/s (1,900Mbps)
File copy speed to FreeNAS, 458.3MB/s (3,365Mbps!!)
iPerf from my iMac to the ProxMox cluster, 10GBe both ends, just under 8Gbps
previous rack stack
previous rack stack
Reverse Proxy
So, there's this thing called a reverse proxy. For the longest time I didn't really know much about them, and didn't think I needed them. I had a bunch of port forwarding rules, and it kind-of-sort-of worked, but all of my addresses where https://bdfserver.com:SomeWeirdPortNumber, which was ugly, and not easy to just simply forward.
Then someone turned me onto nginx reverse proxying. It was a little awkward at first, and getting Let's Encrypt to handle the SSL certificates automatically took me a bit of tinkering, but now it's awesome. And I can give all of my services a subdomain of bdfserver.com and they can happily forward on the port 80/443 of HTTP, or any other port I need, and my router just sends them to the proxy, and the proxy handles the rest. Very cool.
Speeds
I have fiber-to-the-home from Cincinnati Bell (Fioptics Service) and I get most of the advertised speeds as I'm close to their main offices. Here's a typical set of speed tests. My 5k iMac pulling 936Mbps download and 238Mbps upload. The second image is from my iPhone - 418Mbps download and 230Mbps upload. The third image is of a iperf3 performance test from my Plex virtual server to my iMac, 7.28Gbps on the 10Gbps link. I'm not sure why it's only 7+, and maybe jumbo frames would help, but that's very consistent. The fourth image is file transfers to my NAS, the QNAP 12 drive x 10G spinning disks. {UPDATE: I now know why, my USG-XG 10G router is a bit of a bottleneck. I have since added a 10G third networking interface to each of my ProxMox virutal hosts and they all can now get 9+Gbps to each other and my storage SAN.} 325-350MBps is pretty typical, and for spinning disks I think is pretty good. The last image in this set is the iperf from the QNAP to the iMac, at a little over 8Gbps. With a 350 mega BYTE per second file copy speed I can copy a 10GB movie file to the NAS in about 30 seconds. That's nice.
Update: After some tweaking, I'm getting over 400MB/s copies to my VM NAS and 540 MB/s to my file NAS, and better than 8Gbps iperf network speed connection tests between my virtual hosts and my FreeNAS SAN.
Wired internet to DSL Reports
iPhone 7 WiFi Speedtest to MetroNet in Lexington on my Unifi UAP-AC-HD unit upstairs
UPDATED: 9+Gbps from all VM hosts to FreeNAS share
8Gbps VM Host svr-02 to FreeNAS share
539 eye-popping MegaBTYES / second from iMac to QNAP file share (movie data)
450 MB / second from iMac to FreeNAS VM share (Install ISO, VM Images, and Backups)
OLD: 7.28Gpbs iperf Plex VM to iMac
OLD: 354MBps file copy iMac to QNAP
OLD: 8.1Gbps iPerf from QNAP to iMac
Software
ProxMox
I chose ProxMox as my virutalization HyperVisor, at least initially, because it's free and open source, and has most of the features as VM Ware, which I had originally intended to use (with a low cost VM User Group license). It has high availability cluster, live migration from node-to-node of VMs, and although it's not a true Type 1 hypervisor, it sort of runs on the bare medal, with a basic install of Debian at the core, and the ability to run KVM containers as well as full VMs (for OSes like MacOS and Windows) on the same host.
FreeNAS
I had planned to use a Drobo B810i on iSCSI as my main Storage Area Network device for my ProxMox cluster, but it was not able to run an SSH server to allow the cluster to login, so I had to go another route. I chose to use FreeNAS on another Dell server. This is kind of the recommendation of the ProxMox folks, and it's running on Unix (in this case FreeBSD because it has the ZFS file system natively) and allows me to use the ZFS file system, which allows me to use all storage types on a network host that appears to the VMs as local storage. Having "remote" storage allows me to live migrate a VM - move it from one physical server to another while running, without the VM knowing that it is being moved.
Door
The door is an 8' single-lite exterior door (because it is a different climate zone, and the room is loud) with an August smart deadbolt lock, with custom signage. It has an August smart lock on it so I can control and monitor who has access.
Door
August
August
Patch Panel
Here's the final patch panel, with the fiber lines connected as well. We have nearly 96 copper (Cat61) drops and 10 fiber (OM4) runs (2 to each of the offices and to the old utility demarcation). The patch panel separates the house wiring from the rack wiring so that each can be logical for their own purposes. The racks are connected via color-coded patch cables behind a panel inside the wall, to a channel under the floor, and to underneath each rack where they run to in-line Ethernet keystones in another set of patching panels, and then on to the appropriate networking switch.
Final Patch Panel
Patch
Patch
Patch panel with fiber
Fiber
Keystone patch
Keystone patching panel
Patch raceway
Servers
|
|
|
|
|
|
|
|
Network Diagrams
Don't use these to try to hack me, that wouldn't be cool. For those interested.
Before Pictures
I lucked out on the size of the data center, there was this awkward room already that we really didn't know what we were going to do with. It happened to have a window and an exterior wall to cut through for air conditioning, which was nice. It was tall enough, and happened to also connect to a furnace/utility space where we'll mount the electrical panel which will support all of the needs of this room.
You can see that this room has a steel beam running across the short direction, nearly in the middle. My plan, as you can see on the drawing, is to use this to capture the hot air and get it back to the mini split air conditioner. Since the AC will be on the outside wall, this is the cold aisle of the server room, and the door side of the room will be the hot side. I did some basic calculations that said 1 ton of AC (12,000 Btu) is plenty for now, but that I might need 1.5 later, so we're going with that. I got a low ambient Trane model because that's what our HVAC crew mostly installs (brand-wise, I had looked at the Gree and Mitsubishi's as well). It can cool my room even if the outside air temp is below freezing, which is good because the servers will always require cooling.
You can also see from the plan that I mean to have shelving units around the outside walls of the room. That isn't really ideal, but I'm used to having all of the "computer stuff" near by, and with this much room it only made sense. I have complete access to both sides of the racks, and enough room to get closable plastic bins off of the shelves as well. I'll have a small desk on the one wall with a workstation consisting of monitor, keyboard, and mouse hooked up to a USB KVM switch in the rack so I can control the servers. I had a rack-mount monitor before, but it got in the way of doing maintenance, and made the monitor smaller than I would like. All connections to the rack are via under-floor cabling to keep the room tidy. There will also be an HDMI monitor, connected by multi-switcher, to my Tivo DVR units so I can configure/monitor them (or just watch TV!).
Also note that I've designed the room with an elevated floor, with a ramp to the door. I wanted to be able to route power under the racks, with 3 independent circuits under each, with each rack to share 1 circuit with the rack next to it. The idea being that each rack will have a primary and backup UPS for all of the gear that I've since bought that has dual PSU (right now that's a 48 port switch, the UBNT USW-L2-48 and USG-XG-8, and my QNAP NAS, a TVS-1271U-RP-i7-32G). Also all of the Ethernet cabling will run from the patch panel on the wall back inside the wall, under the floor, and to patch panels in each rack. That way the house wiring is completely independent from the rack and a nice clean install.
The Plan - Yes, I'm an engineer in my day job, and this was probably overkill, but you can see how the data center butts up against the crafting-sink-area
We'll square this room off at 11'x17'
This little bump out was just for the cabinets, it'll go away - The tape marks the location of the racks
The previous owner used this as a crafting room - this near area will stay a sink, the space to the right becomes a powder room, the data center is in the background
This is from inside the new Data Center
This is the rack up and running at the previous house - if you look closely you can see the security camera software running
The front of the rack have multi-color LED, which are normally set to a soft blue, just because I think it looks cool
The back of the racks has white LED lighting to make working in there easier - these are normally off
Racks await install at Congress
The rear doors were low-flow nearly closed metal, so I replaced them with new black doors, again, because it looks cooler. Cool air will flow in the white fronts facing the windows, and hot air out the black doors that will face the door to the room
I originally bought 7 rolls of Belden Cat6a cable from Anixter locally for my installation - I wanted to be able to do 10G on the copper - I ended up going back for 2 more and using most of it. I installed dual fiber to each Office space as well, which is 40G capable
Belden 10GXS12 Category 6a cable
Spools of cable await installation
I bought pre-terminated runs of OM4 40G capable armored-jacket fiber optic. I wanted to be able to pull it without worrying too much about damaging the fiber
Network Design
Construction Pictures
With all of that as preliminary, the first thing to do was raise the closet door, and build the floor up. The floor is just one 2"x4" pressure treated stud (this is a basement and could get moist) on it's side, on 12" centers since the rack could be quite heavy, with 3/4" plywood on top. There is a built-in trough for the Ethernet cabling to pass through, with a removable cover so I can get in there and remove or add a single cable at a time if need be. This also allows for a standard size wall-box for the power receptacles under the floor.
I found a local company to pull the Ethernet for me. Anywhere there was going to be a TV I placed a jack at receptacle height as well as at TV height. I also allowed for a few stand-alone monitors - I hope to create a family calendar and information monitor system that will display at a few locations around the house, and display the security camera output. I included literally any place I could imagine a TV or monitor going so I wouldn't regret ignoring it later, including a drop in the garage at bench height for these handy "web relays" that I have used in the past for operating my older style garage door openers and about 10 drops for Ubiquiti POE security cameras.
For offices I pulled 2 Ethernet and 2 fiber to the logical computer location. I pulled dual Ethernet to each WiFi location so that I could power a WAP as well as (potentially) have a 10G connection as I understand the current generation of access points can't be powered off of the same port that passes 10G data. I went with Belden 10GXS12 Cat 6a cable as I wanted to make sure the cable could do the speed, and I could always work with the terminations if I had difficulty with speeds. This cable is huge and stiff, nearly 1/3" each cable with the white plastic spline down the middle.
I wanted to minimize the number of switches required in my infrastructure, but the Cat6a cable is pretty massive, and fairly expensive so I decided 2 runs was the most I could realistically justify at each location. At a minimum, I can have copper going to each computer, and a copper 10G still available for a switch for the balance of the users at that location. Note that my computers (Macs) are the only 10G capable devices I have today, aside from my 2U 12-Bay QNAP NAS which has dual 10G fiber connections and my Ubiquiti 10G USG-XG router and UAS Application Server with dual copper ports, but they are rack mounted (UPDATE: All of my rack-mount Dells now have dual 10G copper ports). I have 3 Ubiquiti 10G USW-XG-16 switches (1 in each rack), and 2 48 port switches (1 USW-48-750W and 1 USW-L2-48 with dual PSU) as the backbone of the network, but all of the remote switches have at best SFP 1G fiber ports. Hopefully Ubiquiti will bring 10G SFP+ to the 8 port switches soon, but the backbone of the network will be dual 10G fiber, with 10G to each of the computers. If the price of the 10G switches comes down, I'll put a 10G USW-XG-16 in each office with dual fiber uplink and a 10G copper to each computer. That's how I ran at my old house, but it's limiting as these switches only have 4 copper ports (UPDATE: UBNT now has reasonably priced copper transceivers for these switches).
For fiber I went with OM4 with LC connectors on each end. I found wall plates and a patch panel for LC connectors, and all of my switches have LC connectors. I went with these 10/40/100G cables with multimode 50/125 micron cable in armored jacket. This might not be perfect, but seemed to give me decent confidence that we could install it safely, and even have potential to move up from 10G later as switches get cheaper. Altogether we pulled about 96 Ethernet and 12 fiber lines.
I also included dual fiber, dual Cat6a, and dual Coax from the current "utility" space where the cable and fiber services enter the house to the new data center. I hope to move those utilities as we finish remodeling the house, but that has yet to be determined (UPDATE: This move was made for the fiber, but the TV still comes into what will be my master closet). Until then I can bring the incoming cable, fiber, attic antenna to data center - over fiber or Ethernet. I plan to rack mount my Tivo base units (UPDATE: Done, check out the tivo rack mount kits under Parts and Bits), with a Cable as well as Over-the-air box feeding 4k Tivo Minis at each TV. My wife and I had cut the cord, but my in-laws aren't ready to do that yet. I also plan to rack mount my Sonos Connect Amps for all of my basement spaces with these handy 3U shelves, part #262-2967 (UPDATE: Done - each room in the basement got a local volume control and dual ceiling speakers, and each TV location has a digital audio back to the rack so I can place the TV sound onto the speakers with a Apple Airport Express).
In terms of users of the network, part of getting this house was moving my in-laws in so they could have some support as they age, and have single-floor living. My wife and I are remodeling the basement for our master suite. The existing house has wired sound, with Sonos users, and it's a large rambling house with a pool and cabana, so I have dual Ubiquiti HD wireless APs on each end of the main house, 1 in the garage and pool cabana, and 1 each upstairs and in the basement. There will be 1 as well in the new garage we will be adding.
I have installed a bunch of HomeKit automation gear. I have installed a lot of Leviton Decora Smart Switches to control the house lighting, the fountain, Eve motion detectors and door/window modules, degree temperature sensors, iDevices outdoor switches for outdoor lighting, 2 Nest thermostats, 3 August locks with WiFi extenders, a Rain Bird WiFi sprinkler module, 5 Sonos Connect Amps (and I plan to add several more for the basement remodel-UPDATE: Done), 2 laser printers, 2 multi-function printer, copier, scanners, 5 Ring flood light cameras and 3 Ring doorbell cameras, and 16 Ubiquiti cameras (some POE and some micros on WiFi). I have 2 work laptops in the house, 2 personal laptops, 2 desktops, 1 windows server, the UAS, the QNAP NAS, and I plan to add some servers (UPDATE: Done, see above), maybe Dell 720s. I have about 10 Tivo devices, and nearly as many Apple TVs (we do a lot of Airplaying) as well as a HomePod, 4 iPhones, 5 Echos, and 4 Kindles and about 6 "Smart" TVs. Also the Security system will be upgraded to allow WiFi connectivity. At any moment I have nearly 150 WiFi devices.
Nearly as important as having all of this connectivity is keeping it straight, well understood, and maintainable. To that end, I have clear Brother labels on each wall plate with the patch panel number, and each wire has a heat-shrink label with a clear heat shrink protective covering layer to keep it legible. Each cable has a four digit number. The first digit is the patch strip (0/1/2/etc) and then the port number, so "0001" for the first port in first patch panel. The house patch will be connected, by routing back inside the wall, to an inline patch in the respective rack it's served by. The house patch panels are these 24 port Cat6a units - giving me 96 ports for Ethernet, and the wall mount rack is the StarTech 6U "WallMount6". The fiber patch panel is this 12 port unit. For the in-rack connections I'm dropping these SF in-line couplers from the wall patch to feed the switches. I plane to have a patch above and below each 48-port switch to keep the connections nice and tidy. Each Power circiuit is similarly labelled, and the power cables in the data center are color-code and labelled. Each UPS is labelled with the circuit that feeds it, and each server is labelled with the UPS that servers it.
I'll post my network diagram soon (UPDATE: Done), but here are some pictures midway through construction. As of June 25, 2018 the wire is pulled, tested, and in the jacks. Since there's so much construction, I have a temporary network setup inside the closet, so the patch is kind of nasty looking, with wires poking through a small hole in the rear drywall. Long term I'll clean up this wire, and it will get a OSB or melamine cover that will be painted to match the wall color (UPDATE: Done), and all of the patch cables to the rack will likewise come off the patch and look inside the way, under the flooring, and up into the rack. The floor will be carpeted with low pile commercial carpet squares from FLOR called Lasting Grateness. By the way, all of my 10G NICs for my Macs are Sonnet Twin 10G Thunderbolt 2 units.
As I think I mentioned, I went with a Trane 1.5 ton (18,000 Btu) low-ambient (can cool below freezing outside air temp) mini-split AC unit. This unit can heat, but I shouldn't need that. I will have them mount a hard-wired thermostat on the hot-side wall to verify cooling rather than depend on the wireless handheld unit, which might not be able to see over the 7' tall racks. As you can see, we're framing in for an 8' exterior door to hold the cold in, and provide perimeter security. With the whole-house monitored security and video system this should be adequate.
For the wire pulling I hired Totten wiring services to pull my cable. He's a magician at finding ways to get wire where it needs to go.
Awhile back I bought a Triplett "Real World Certifier" for testing cables. It does a pretty good job, and will show you if you have any breaks or shorts, and verify the wire order and give basic speed capability and an overall "Cat" score on the scale of Cat3-Cat6. It's not a real cable tester, and Jeff will be bringing the real deal over to certify my cables, but it's was a good check before putting some of the lines into temporary service.
Workstation - Monitors for Tivo and serve control
In-Line Ethernet Coupler
Fiber patch panel
Cat6a Patch Panel
6U Wall Mount Rack for patching
Flor "Lasting Grateness" in Indigo
Sonnet Technologies Dual Port 10 Gigabit Ethernet Thunderbolt 2 NIC
2U Rack mount for 3 Sonos Connect Amps
Armored fiber cable
Heat shrink wire label with clear heat shrink cover
Multi-line labels are easier to read from any side
Lots of wire pulled, empty spools in background
Main patch panel is up and running, but messy
Many, many runs completed
New lighting to be seen, the beam soffit will trap hot air on rack side of room
This 8' opening for rolling racks in and out of data center - the inset behind the door is for the ramp, which will get shoe molding to keep you from walking off the edge
Door framed and electric at workstation/desk location
Mini split AC compressor
Mini split outside unit mounted above the snow line so it cool data center all winter long
Mini split indoor unit on cold side of data center
Panel
Panel
DSL Reports Speedtest on USG-XG-8 with Full Intrusion Prevention enabled FTTH (Cincinnati Bell Fioptics)
The racks await the room
Circuits mounted in floor await the racks
AC
AC
Labels everywhere to keep things straight
Fiber wall plates
Fixing the drywall and installing the lighting
Power outlets in floor
Floor sheathed
Floor removable panels are evident
Carpeting installed
Speaker volume control
Speaker wire
Installing the racks
Getting the racks ready
Patching the racks up
Under-Server rack grommets and running wire inside the rack side panel as it doesn't have vertical cable management
Patching the rack and testing Coax connections
In-wall patch routing
Rear of patch panel/in-line Ethernet coupler
Starting to patch up to the house wiring
Painting the room
Shelves installed - note that the bin fits between racks and shelves
Optical Network Interface before relocation
The data center is ready to go - the painted wood panel in far right is where my test stand will be for new networking gear, and the top has my UPS for my ONT and cable modem
Temporary network and server in closet of "future" data center during construction
Parts and Bits
Here are the bits and bytes in my Data Center. I apologize that it's a bit rambling, I'll try to make it more coherent later, but this is meant to list the makes, models, etc. of hardware devices in my installation.
Most of the Networking gear is Ubiquiti/UBNT.com stuff running UniFi on a UAS-the UniFi Application Server, or what is now called the UAS-XG Server. It's a Xeon class server with 32GB of ram, and I've replaced the stock video storage to dual 10G Seagate Iron Wolf Pro drives. And the network side it has dual 10G copper ports in a 1U rack mount chassis.
The router is the new UniFi Security Gateway USG-XG-8 with 8 SFP+ ports and 1 10G copper port and 80Gbps routing capability, while doing full IPS/IDS at 1G speeds. It has 16 cores and 16GB of ram in a 1U dual-PSU rack mount chassis. I really only got this so that I could do the IPS/IDS at full 10G like speed between my VLANs. The network is fed by Fiber-to-the-Home/FTTH by Cincinnati Bell Fioptics at 1 Gbps download/250 Mbps upload through an Alcatel-Lucent 7342 ONT, or Optical Network Terminal. I'd love to hook this directly to my router, but they only support GPON, not SFP, and don't support customer equipment for the ONT, though they do allow you to bring your own router.
The core switches are all USW-XG-16 with 12 SFP+ ports and 4 10G copper Ethernet ports. Each rack with have one of these switches connected to the primary by dual 10G multimode fiber patch cables. Each of these switches will support 10G copper links to my servers and 1G or 10G links to my UniFi switches. Currently only the 48 port UniFi switches have 10G SFP+ ports.
My primary distribution switch will be the as-yet-unreleased UniFi 48 port L2 POE switch with dual PSU, the USW-L2-48-POE pair with a USW-48-750W POE switch, giving me 96 ports of POE goodness.
I have various other UniFi switches, including a USW US-16-150W in the rack, various USW-8-150W at the end points (mostly offices) - each of with can accept 1G fiber via SFP, and a bunch of USW-US-8 units, mostly at TVs and gaming consoles. For WiFi I recently swapped out my UAP-AC-Pro units for UAP-AC-HD ones. The HD is slightly bigger than the Pro, but has a lot better throughput. I can get nearly 400 Mbps both directions on my iPhone on the HD, which is about double what I got on the Pros. Also, the beauty of the access points and US-8 switches are that, like the cameras, they're all POE. That means that when my power goes out, these can stay up as each rack is protected with 1 hour of UPS backup. So, as long as I have light on the incoming fiber, I have networking! I currently have 6 of the UAP-AC-HD mounted, 1 on each wing of the first floor, 1 in the garage at ground level, 1 in the basement and 1 on the second floor in the center of the house, and finally one in the pool cabana to cover the backyard entertaining space.
My backup power, and power line conditioning, is by CyberPower OR1500LCDRM1U 1U tall 1,500 VA units in each rack, with dual units in the primary rack as many of those devices have dual PSU. Since these only have 4 protected outlets, I have each powering a Cyber Power CPS1220RMS 1U power strip which support the bulk of power users. Another handy feature of these UPS units, they have an environmental module you can connect to them to monitor temperature in the rack. I've recently started playing with the CyberPower PDU20M2F10R mointored PDU. For starters I'll use these to monitor the total power on the 3 circuits powering my servers, and i might add one for each of the UPS units, and I'm considering more UPS units for a couple of reasons. First, I'm not getting the run time I would like, and second I want to setup the Dell virtualization servers with a USB connect each to a UPS so they can get the shutdown message to trigger powering down the guest VMs.
My QNAP network attached storage/NAS, a TVS-1271U-RP-i7-32 that I purchased from SPAN.com, will connect directly to the switch with dual 10G fiber. This 2U rackmount NAS has an Intel Core i7 processor and 32GB of ram and 12 SAS/ drive bays, and I've opted for the dual 128MB (IBQ-XRF256) M.2 SSD cache and dual SFP+ expansion (LAN-10G2SF-MLX) cards. I paired this with 12 Seagate Iron Wolf Pro enterprise/NAS-ready 10G SATA (ST10000NE0004) spinning hard drives in a RAID6 array (2 drives for redundancy, so 100TB of storage). On this NAS I run a Plex server with my entire TV show and movie library as well my music and home videos. I also run a VM with a NextCloud instance for all of my personal cloud needs, and use it as an Apple Time Machine target.
I also have a Mac Mini, 2012 model with quad core i7 and 16GB ram running OSX server edition as my web and email server. It has dual 1TB SSD inside the case, and it mounted in a Sonnet 1U "Rack Mini" chassis. It is connected to the network via a Sonnet Twin10G thunderbolt NIC to the copper port on the USW-XG-16. My main workstation is a 5k iMac, also with a 10G Sonnet NIC wired to the USW-XG-16.
The Plex server is mostly used as a host for all of my Tivo gear, mostly the new Tivo Mini 4k VOX units at the TVs, with Tivo Bolt units in the rack. The nice thing about the newer Tivo units is they all have 1G Ethernet ports. The bad thing about the Bolt is the silly up-bend makes it too tall for a 1U rack shelf! I have a cable card version with 6 tuners as well as an over-the-air antenna version. Each TV in the house also has and Apple TV for AirPlay capability and other uses. The primary TVs have either the 4th or 5th generation units, with the balance just being generation 3 models. The 2 newer generations support Plex and other applications, which can be handy.
The entertainment gear, all in rack 3, finishes out with Sonos Connect Amps, in these cool Penn-Elcom rack mounts. Each room will be wired with in-ceiling speakers, a local volume control, and be wired back to Sonos Connect Amp Zone Player. These are cool because they're network-attached amps that can drive the speakers, and can be controlled by pretty much any device on the network. Even better, you can easily group and ungroup multiple zones to play the same content (local, Apple Music, Plex, Air Port Express Airplay content via line-in) or different content. Finally I have a PS4 and a PS3 at different locations for gaming.
For security I have the advertised "Ring of protection", with a Ring doorbell camera and 5 Ring flood light cameras - all on WiFi. Inside of that I have a monitored Honeywell (Stanley Security) wired home security system with door and window sensors, motion and glass breakage, smoke and CO detection. I'm looking to upgrade this with a full LCD touchpad version with cellular and internet connectivity with iOS app support. I also have wired/POE UBNT cameras monitoring all of the indoor and outdoor spaces and recording to the UAS. I have a mix of the new UAP-G3-Pro and older UVC-G3 cameras on the outside and UVC-Micros covering the indoor spaces on WiFi. The Pro is waterproof and has optical zoom, so it's the workhorse, with covered areas and doorways getting the regular G3 with IR extender.
For other home automation tasks, I have a HomePod and several Alexas (although I may get rid of these soon). I have the Rain Bird WiFi Module for my sprinklers, 5 Nest smart thermostats (although we're considering going to geothermal, so these might have to change to Ecobee units), 3 August Smart Locks. One thing I like about most of the home automation gear I buy is that it can function in dumb mode as well. For instance, the Nest thermostat is intuitive to use as a local thermostat to my in-laws. The August lock from the either side can function as just a keyed deadbolt lock if power or connectivity is lost. My Leviton light switch connect to standard wire and function standardly without connectivity.
I have a bunch of HomeKit gear as well, and I love telling Siri on my Apple Watch to do things. I tried the Hue bulbs, but didn't like that mode of interaction (having to leave the switch on all of the time). Instead I've moved on to the Leviton Decora Smart Switch, homekit edition. These are basically just regular switches that have had WiFi added to them, so if data and/or automation fails you can still turn on your lights. I'm considering Lutron Serena smart blinds as well. For automation purposes I've paired the light switches with Elgato Eve Degree temperature and Motion sensors, and the Eve button. In the winter my mother-in-law's office gets chilly. She can use the Eve Button to trigger a iDevices smart wall switch to turn on her space heater, or the Eve Degree can do it automatically to maintain the right local temperature. I have iDevices Indoor and outdoor switches. The outdoor switches control the pump to my fountain, the lights illuminating my fountain, the string lights over my pool, and the bug zapper next to it.
UPDATE: I have started upgrading my on site servers with Dell rack mount units. Partly this is driven by my desire to get better performance and mvoe to virutal servers, and partly due to Apple deprecating many of the OSX (now MacOS) Server features like web and email servers. Previously I'd run Plex on my QNAP NAS, and got occasional complaints about its ability to transcode video. Also, I run a NextCloud private cloud instance on the QNAP in a VM, and although it has an i7 and 32GB of ram, the performance has not been great. So, this move will allow me to dedicate significant compute resources affordable to easy-to-manage, separate, virtual machines - mostly running linux. I have a pair of Riser 8 port KVM switches to bring the consoles to my local desktop. I've actually also got a 2-1 KVM in the rack so I can switch between the two 8 ports, since I only have 1 keyboard, video, and mouse on my desk. I didn't find a 16 port that had VGA, which is what I'm currently using, and I already had 1 8 port.
My current plan is to split the workload into 3 pieces.
- 1U Dell R620 (ProxMox)
- A thin server with fewer disk slots for less data-intensive workloads
- Domain Name Server (Debian)
- Network Time Server (Debian)
- PiHole Anti-Adware Server (Rasbian)
- Dual Home Assistant instances (Hass.IO) Production/Development
- 2U Dell R720
- Thicker "main" server (ProxMox)
- Email Server (SMTP/IMAP Dovecot/RoundCube)
- Web/Apache Server
- MediaWiki Server
- WordPress Server
- TeamViewer Target/JumpBox
- 2U Dell R720xd (FreeNAS)
- Thicker "SAN" server - Storage Area Network
- Remote storage allows Proxmox "live" migration of Virtual Machines between ProxMox Nodes
- 1 Storage Array of 6 drives (36TB) for Live VM data
- 1 Storage Array of 6 drives (72TB) for VM Backups
Network Attached Storage/QNAP TVS-1271u
1U Rack Server Dell R620 with Bezel
1U Rack Server Dell R620 without its Bezel
2U Rack Server Dell R720 with Bezel
2U Rack Server Dell R720 without its Bezel
2U Rack Server Dell R720xd with Bezel
2U Rack Server Dell R720xd without its Bezel
UAS-Running UniFi Controller and UniFi Video Server
UAS
Router-USG-XG-8 10G capable
XG-8
USW-XG-16 core switches, 1 per rack
Setting up a 20G backbone
Multimode OM4 10G/40G LC-LC Patching Cables
USW-L2-48 and USW-48-750W will be my primary distribution switches
Dual PSU 48 Port UniFi Switch front
Dual PSU 48 Port UniFi Switch back
FTTH ONT-Optical Network Terminal
1 Gpbs download via Optical to Ethernet Terminal (single GPON connector)
16 port POE switch, the USW-US-16-150W
WiFi via UAP-AC-HD
WiFi - old UAP-AC-Pro units replaced
150W POE local 8 port switch, the USW-US-8-150W
POE powered local 8 port switch, the USW-US-8
CyberPower OR1500LCDRM1U UPS front
CyberPower OR1500LCDRM1U UPS back
CyberPower CPS1220RMS power strip front
CyberPower CPS1220RMS power strip front
CyberPower RMCARD205 Networking Card
CyberPower ENVIROSENSOR temperature sensor
CyberPower PR1500LCDRT2U
Color-Coded power cables
Power Monitoring
CyberPower PDU20M2F10R
CyberPower PDU20M2F10R rear
Triplett Real World Certifier RWC1000
2012 Mac Mini running OSX Server
Sonnet Rack Mini Front
Sonnet Rack Mini Back
Sonnet Twin10G Thunderbolt 1/2 NIC for all Mac Computers
Sonnet Twin10G NIC
5k iMac 32G
QNAP NAS TVS-1271U-RP-i7-32-US rack mount NAS
TVS-1271u
IBQ-XRF256 QNAP dual M.2 256GB SSD Cache card
IBQ-X10GM QNAP dual SFP+ NIC
12 Seagate Iron Wolf Pro NAS-ready 10G drives ST10000NE0004
Tivo Bolt Cable 4k
Tivo Bolt 4k
Tivo Mini Vox 4k
Penn-Elcom R1498/3UK-SONOSZP120 Connect Amp Rack Mount
Empty Sonos Rack Shelf
Sonos Connect Amp front
Sonos Connect Amp front
Tripp Lite 12 port Keystone N062-0120KJ for Audio Terminations
Keystone binding posts for audio
Patch panel nearly complete
Airport Express for line-in audio to Sonos Connect
Generation 5 Apple TV 4k
Generation 4 Apple TV with Siri Remote
Generation 3 Apple TV with silver remote
Ring Doorbell Pro
Ring Floodlight Security Camera
UBNT G3 Micro security camera
UBNT G3 security camera with IR range extender
UBNT G3 Pro security camera
Apple Home Pod
Amazon Alexa
Rain Bird sprinkler controller Wi-Fi module
Nest smart thermostat generation 2
August smart lock
Leviton Decora HomeKit switches
Eve Button HomeKit
Eve Degree HomeKit
Eve Motion HomeKit
iDevices Outdoor Switch Homekit
Outdoor Switch - Fountain
iDevices Indoor Switch HomeKit
The Plan - Yes, I'm an engineer in my day job
Great Lakes Racks GL840S2-2436
In-Line Ethernet Coupler
Cat6a Patch Panel
Fiber patch panel
6U Wall Mount Rack for patching
9 rolls of Belden Cat6a cable ended up getting bought, and mostly used
Belden 10GXS12 Category 6a cable
Spools of cable await installation
I bought pre-terminated runs of OM4 40G capable armored-jacket fiber optic.
Fan Unit Model: 40MAQB18B--331--
Compessor Unit Model: 38MAQB18R--301--
Trane Mini split AC
Mini split outside unit mounted above the snow line so it cool data center all winter long
Mini split indoor unit on cold side of data center
Configuration
-All View
Interior
Exterior
Entertainment Rack - Tivo DVRs, Sonos connect amps for house in-ceiling audio
Bottom of Entertainment Rack
Server rack - Running virtual servers for private cloud, web hosting, email, security camera monitoring and recording, etc.
Servers nearly complete - at least for my current plans 2-620, 2-R720, 1-R720XD, 1 QNAP TVS-1271u
In-progress before getting additional servers
Networking rack
Bottom of Networking rack
Back of racks
Front of racks
Code Red!
"Cold Aisle" or front of rack near HVAC unit
Entertainment Rack
Server rack before buying Dell rack mounts, In-progress before getting additional servers
Server rack with new Dell rack mount servers, In-progress before getting additional servers
Networking rack
KVM Switch
KVM Switch
Dual 8 port KVM switches for server monitoring
Monitored PDU
