Camp Kiwi Data Center: Difference between revisions
| Line 130: | Line 130: | ||
{| cellspacing="0" cellpadding="10" | {| cellspacing="0" cellpadding="10" | ||
|style="width: 20%"|<hovergallery maxhoverwidth=1200 maxhoverheight=1024> | |style="width: 20%"|<hovergallery maxhoverwidth=1200 maxhoverheight=1024> | ||
Image:tvs-1271u-RP.png | Image:tvs-1271u-RP.png|TVS-1271u | ||
</hovergallery> | </hovergallery> | ||
|style="width: 80%"| | |style="width: 80%"| | ||
| Line 162: | Line 162: | ||
|- | |- | ||
|style="width: 20%"|<hovergallery maxhoverwidth=1200 maxhoverheight=1024> | |style="width: 20%"|<hovergallery maxhoverwidth=1200 maxhoverheight=1024> | ||
Image:R720_bezel.jpg | Image:R720_bezel.jpg|R720 | ||
Image:R720-nobezel.jpg | Image:R720-nobezel.jpg|R720 | ||
</hovergallery> | </hovergallery> | ||
|style="width: 80%"| | |style="width: 80%"| | ||
| Line 181: | Line 181: | ||
|- | |- | ||
|style="width: 20%"|<hovergallery maxhoverwidth=1200 maxhoverheight=1024> | |style="width: 20%"|<hovergallery maxhoverwidth=1200 maxhoverheight=1024> | ||
Image:R720XD_bezel.jpg | Image:R720XD_bezel.jpg|R720XD | ||
Image:R720XD-nobezel.jpg | Image:R720XD-nobezel.jpg|R720XD | ||
</hovergallery> | </hovergallery> | ||
|style="width: 80%"| | |style="width: 80%"| | ||
Revision as of 09:22, 17 January 2019
Introduction
I started my "home lab" experience at our home on Wandering Way, by pulling a couple of Ethernet cables from one side of the house, where the cable came into the building, to my office. I also pulled one from there to a WiFi access point on the other side, and a computer in the kitchen island.
From there we moved to Top Flite where I converted a pretty good space (maybe 10'x8'?) into a mini data center. My brother-in-law is in the surplus business and ended up with some old Great Lakes 45U racks he didn't need, so I picked one up. I actually picked up 3, but I only used one over there. It's not ideal, being a "Security and Sound" GL840S2-2436 rack, it doesn't have any cable management features, but I got them basically free (thanks Steve!), so who's arguing?
That Top Flite house was built in the early 2000's, and was wired with Cat5e wire to most of it's rooms, alongside dual RG6 coax, I think for the phone lines. These were homerun back to the utility space that I'd taken over for the data center, so I started getting rack-mount switches, routers, and shelves and mounted my cable model and mac mini server in there. I upgraded to a USG Pro and USW-16, and eventually a USW-48-750 to power my WiFi access points and some cameras. I became an amateur wire-puller as I added ceiling-mounted UAP-AC-Pros in 4 places around the house and started to monitor the property with Foscam PTZ cameras, and added dual fiber drops to a couple of computer locations.
Eventually I added a portable 1 ton AC unit and QNAP NAS for a Plex server, and I was really a Home Labber! The heat from the AC was ducted to the attached garage, which as luck would have it was on the adjoining wall. Somewhere along the way, Cincinnati Bell started offering 1 Gbps FTTH service (250 Mbps upload) and I snapped that up too. That upload speed was particularly important for my Plex server.
Then it came time to move again. I got my wife to agree that we would only look at houses that had fiber service, and that I would get a budget to install good Ethernet cabling. And so it began...
Stats
The data center features Ubiquiti Unifi "pro-sumer' networking gear, with:
- Nearly 90 Cat6a copper network runs, and a mile and half of cable
- 10 OM4 fiber-optic network runs (capable of 40 or 100 gigabit per second)
- 8-20A power circuits
- 1.5 tons of AC cooling
- Core network is dual 10 gigabit per second fiber optic
- Primary internet service to house 1 gigabit per second fiber optic
- Backup internet service to house 200 megabit per second cable
- File server is 120 terabytes of disk space (running Plex and Nextcloud personal cloud service)
- 1U Virtualization Server is Dell R620 (local services, DNS, NTP, Ad Blocking, Home Automation)
- 2U Virtualization Server is Dell R720 (Cloud Data Server, Movie Streaming, eMail, Web, Wiki)
- 2U Storage Area Network Server is Dell R720XD (6x6TB VM storage, 6x12TB Backup)
- 7 Sonos network audio amps for basement level audio
- 4 Sonos network audio amps for main floor audio
- 3 Tivo DVRs for entertainment, streamed to 6 different locations
- 12 camera security system with on site video backup
Origin of the Name
There's this guy named "Dave". I used to work with this guy, and he is a bit... eccentric. Anyway, he managed a Data Center at the office, and had some folks supporting it. Some of these folks were not native English speakers, and he had a Monday morning telephone conversation where it got confused as to what they were saying about eating a can of peaches versus visiting "Camp Peaches", presumably like a summer camp for Scouts. In a fit of brilliance he decided to name his Data Center Camp Peaches. He even acquired signage to proclaim same. When he took on a new role, I inherited this sign (our IT organization did not see the brilliance in the naming scheme - however, it is still universally known corporately as Camp Peaches).
Camp Peaches, origin of the 'Camp Kiwi' moniker
Finished Pictures
Here are some snaps of the finished product. I'll be adding these as each section comes online. This section is an overview, the details of everything are below in the 'parts and bits' section.
Workstation
The basic Data Center is now online. As you see in the below pictures, I have a small workstation inside the data center. This is not my office, I have a full office elsewhere - and it's too cold and noisy in the data center to spend a lot of time there. But, I have a KVM switch connected to my QNAP NAS, Mac Mini webserver, and the UniFi Application server to allow me to interface with these servers. The workstation allows me to not only administer the servers, but also monitor the security cameras from around the house. I also have a monitor connected to a HDMI switcher that allows me to monitor each of my Tivo Bolt DVRs and validate they are functioning properly.
Workstation - Monitors for Tivo and server control
Security Camera Configuration
Cameras-All View
Cameras Interior
Cameras Exterior
The Design
Climate Control
Data center temperature is managed with a 1.5 ton Carrier mini split AC unit. This unit is capable of cooling the data center all winter with outside air temperatures well below freezing. The wall-mounted thermostat is on the wall in the hot-aisle to check the server exhaust air temperature, and is set to keep that temperature in the 60s Fahrenheit. Also, the temperature is monitored continual with an Sensor Push temperature probe on a shelf in the hot aisle, as well as CyberPower UPS environment monitors on the top of each rack. Both of these device types can SMS/push notify me if the temperature or humidity gets out of range.
HVAC
Mini split AC compressor
Mini split outside unit mounted above the snow line so it cool data center all winter long
Mini split indoor unit on cold side of data center
Thermostat
Thermostat final
Power
A new subpanel was mounted adjacent to the data center to provide power to the server space and the new AC unit. This panel provides eight 20A circuits to the data center below the raised flooring. 2 circuits are under the networking rack providing primary and backup power, and 3 circuits to each server cabinet. Each rack is provided with a CyberPower 1500VA UPS, with the networking and server racks having both primary and secondary/backup units.
Power Outlets in Floor under racks
Subpanel
Flooring
First, the floor was raised 4" (pressure treated 2"x4" studs on edge) and placed on 12" centers for strength with 3/4" plywood covering. Within this flooring, in addition to power, are trenches with removable covers for routing data cabling from the racks to the patch panel as well as KVM and HDMI wiring to the workstation. The floor is covered with Flor carpet squares which are thin and easy to roll a rack across and durable, but also attractive. The tiles are just under 20" square (50cm x 50cm), and the style we picked was "Lasting Grateness" in indigo and bone, simulating an iron grate.
Power Outlets in Floor under racks
Carpet Design
Wiring trench in floor
Open trench before wire
Trench nearly full
Everything but the fiber optics
Servers and Racks
I have 3 racks for my equipment. The first rack houses all of the networking gear, the middle rack my servers, and the last rack has the entertainment hardware like Sonos sounds systems and Tivo networked DVR (digital video recorders for cable and over-the-air TV). Each rack has backup battery power in the event of electrical outage and lighting. The fronts feature RGB LED "mood" lighting, and backs have white LED lighting for task illumination, which is usually off.
I have a few servers. The most impressive of which is a QNAP 12 bay NAS, or 'network attached storage'. This has PC class CPU and twelve 10 Terabyte hard drives, two of which serve for redundancy in case of failure, so I net out at around 100 TB of total storage. This computer hosts my plex server and library as well as a virtualization environment that runs a private cloud (NextCloud) instance. This cloud keeps the family's files and picture, replicating them across our devices, as well as a shared family calendar and contacts/address book. The Plex content is available on our Tivos as well as iOS, Playstation, and other devices in home and away.
The other main server that I host is my mail, web, and Wiki server, ferrellmac.com running on a Mac Mini. Apple is abandoning the Server product, so these will soon move to an Open Source Linux system.
My final current server is a Ubiquiti UniFi Application Server. This is a purpose-built system in the UniFi line that hosts both the UniFi "software defined network" and the UniFi video security camera NVR (network video recorder). This recorder supports our 13 security cameras.
"Cold Aisle" or front of rack near mini-split HVAC unit
"Hot Aisle" - Back of racks (white LED lighting provided for service)
Front of racks (multi-colored LED for mood lighting)
Entertainment Rack - Tivo DVRs, Sonos connect amps for house in-ceiling audio
Server rack - Running virtual servers for private cloud, web hosting, email, security camera monitoring and recording, etc.
Networking rack
Software
ProxMox
I chose ProxMox as my virutalization HyperVisor, at least initially, because it's free and open source, and has most of the features as VM Ware, which I had originally intended to use (with a low cost VM User Group license). It has high availability cluster, live migration from node-to-node of VMs, and although it's not really a type 1 hypervisor, it sort of runs on the bare medal, with a basic install of Debina at the core, and the ability to run KVM containers as well as full VMs (for OSes like MacOS and Windows) on the same host.
FreeNAS
I had planned to use a Drobo B810i on iSCSI as my main Storage Area Network device for my ProxMox cluster, but it was not able to run an SSH server to allow the cluster to login, so I had to go another route. I chose to use FreeNAS on another Dell server. This is kind of the recommendation of the ProxMox folks, and it's running on Unix (in this case FreeBSD because it has the ZFS file system natively) and allows me to use the ZFS file system, which allows me to use all storage types on a network host that appears to the VMs as local storage. Having "remote" storage allows me to live migrate a VM - move it from one physical server to another while running, without the VM knowing that it is being moved.
Door
The door is an 8' single-lite exterior door with an August smart deadbolt lock, with custom signage. It has an August smart lock on it so I can control and monitor who has access.
Door
August
August
Patch Panel
Here's the final patch panel, with the fiber lines connected as well. We have nearly 80 copper drops and 10 fiber runs (2 to each of the offices and to the old utility demarcation). The patch panel separates the house wiring from the rack wiring so that each can be logical for their own purposes.
Patch
Patch
Patch panel with fiber
Fiber
Servers
|
|
|
|
|
|
|
|
Before Pictures
I lucked out on the size of the data center, there was this awkward room already that we really didn't know what we were going to do with. It happened to have a window and an exterior wall to cut through for air conditioning, which was nice. It was tall enough, and happened to also connect to a furnace/utility space where we'll mount the electrical panel which will support all of the needs of this room.
You can see that this room has a steel beam running across the short direction, nearly in the middle. My plan, as you can see on the drawing, is to use this to capture the hot air and get it back to the mini split air conditioner. Since the AC will be on the outside wall, this is the cold aisle of the server room, and the door side of the room will be the hot side. I did some basic calculations that said 1 ton of AC (12,000 Btu) is plenty for now, but that I might need 1.5 later, so we're going with that. I got a low ambient Trane model because that's what our HVAC crew mostly installs (brand-wise, I had looked at the Gree and Mitsubishi's as well). It can cool my room even if the outside air temp is below freezing, which is good because the servers will always require cooling.
You can also see from the plan that I mean to have shelving units around the outside walls of the room. That isn't really ideal, but I'm used to having all of the "computer stuff" near by, and with this much room it only made sense. I have complete access to both sides of the racks, and enough room to get closable plastic bins off of the shelves as well. I'll have a small desk on the one wall with a workstation consisting of monitor, keyboard, and mouse hooked up to a USB KVM switch in the rack so I can control the servers. I had a rack-mount monitor before, but it got in the way of doing maintenance, and made the monitor smaller than I would like. All connections to the rack are via under-floor cabling to keep the room tidy. There will also be an HDMI monitor, connected by multi-switcher, to my Tivo DVR units so I can configure/monitor them (or just watch TV!).
Also note that I've designed the room with an elevated floor, with a ramp to the door. I wanted to be able to route power under the racks, with 3 independent circuits under each, with each rack to share 1 circuit with the rack next to it. The idea being that each rack will have a primary and backup UPS for all of the gear that I've since bought that has dual PSU (right now that's a 48 port switch, the UBNT USW-L2-48 and USG-XG-8, and my QNAP NAS, a TVS-1271U-RP-i7-32G). Also all of the Ethernet cabling will run from the patch panel on the wall back inside the wall, under the floor, and to patch panels in each rack. That way the house wiring is completely independent from the rack and a nice clean install.
The Plan - Yes, I'm an engineer in my day job, and this was probably overkill, but you can see how the data center butts up against the crafting-sink-area
We'll square this room off at 11'x17'
This little bump out was just for the cabinets, it'll go away - The tape marks the location of the racks
The previous owner used this as a crafting room - this near area will stay a sink, the space to the right becomes a powder room, the data center is in the background
This is from inside the new Data Center
This is the rack up and running at the previous house - if you look closely you can see the security camera software running
The front of the rack have multi-color LED, which are normally set to a soft blue, just because I think it looks cool
The back of the racks has white LED lighting to make working in there easier - these are normally off
Racks await install at Congress
The rear doors were low-flow nearly closed metal, so I replaced them with new black doors, again, because it looks cooler. Cool air will flow in the white fronts facing the windows, and hot air out the black doors that will face the door to the room
I originally bought 7 rolls of Belden Cat6a cable from Anixter locally for my installation - I wanted to be able to do 10G on the copper - I ended up going back for 2 more and using most of it. I installed dual fiber to each Office space as well, which is 40G capable
Belden 10GXS12 Category 6a cable
Spools of cable await installation
I bought pre-terminated runs of OM4 40G capable armored-jacket fiber optic. I wanted to be able to pull it without worrying too much about damaging the fiber
Network Design
Construction Pictures
With all of that as preliminary, the first thing to do was raise the closet door, and build the floor up. The floor is just one 2"x4" pressure treated stud (this is a basement and could get moist) on it's side, on 12" centers since the rack could be quite heavy, with 3/4" plywood on top. There is a built-in trough for the Ethernet cabling to pass through, with a removable cover so I can get in there and remove or add a single cable at a time if need be. This also allows for a standard size wall-box for the power receptacles under the floor.
I found a local company to pull the Ethernet for me. Anywhere there was going to be a TV I placed a jack at receptacle height as well as at TV height. I also allowed for a few stand-alone monitors - I hope to create a family calendar and information monitor system that will display at a few locations around the house, and display the security camera output. I included literally any place I could imagine a TV or monitor going so I wouldn't regret ignoring it later, including a drop in the garage at bench height for these handy "web relays" that I have used in the past for operating my older style garage door openers and about 10 drops for Ubiquiti POE security cameras.
For offices I pulled 2 Ethernet and 2 fiber to the logical computer location. I pulled dual Ethernet to each WiFi location so that I could power a WAP as well as (potentially) have a 10G connection as I understand the current generation of access points can't be powered off of the same port that passes 10G data. I went with Belden 10GXS12 Cat 6a cable as I wanted to make sure the cable could do the speed, and I could always work with the terminations if I had difficulty with speeds. This cable is huge and stiff, nearly 1/3" each cable with the white plastic spline down the middle.
I wanted to minimize the number of switches required in my infrastructure, but the Cat6a cable is pretty massive, and fairly expensive so I decided 2 runs was the most I could realistically justify at each location. At a minimum, I can have copper going to each computer, and a copper 10G still available for a switch for the balance of the users at that location. Note that my computers (Macs) are the only 10G capable devices I have today, aside from my 2U 12-Bay QNAP NAS which has dual 10G fiber connections and my Ubiquiti 10G USG-XG router and UAS Application Server with dual copper ports, but they are rack mounted. I have 3 Ubiquiti 10G USW-XG-16 switches (1 in each rack), and 2 48 port switches (1 USW-48-750W and 1 USW-L2-48 with dual PSU) as the backbone of the network, but all of the remote switches have at best SFP 1G fiber ports. Hopefully Ubiquiti will bring 10G SFP+ to the 8 port switches soon, but the backbone of the network will be dual 10G fiber, with 10G to each of the computers. If the price of the 10G switches comes down, I'll put a 10G USW-XG-16 in each office with dual fiber uplink and a 10G copper to each computer. That's how I ran at my old house, but it's limiting as these switches only have 4 copper ports.
For fiber I went with OM4 with LC connectors on each end. I found wall plates and a patch panel for LC connectors, and all of my switches have LC connectors. I went with these 10/40/100G cables with multimode 50/125 micron cable in armored jacket. This might not be perfect, but seemed to give me decent confidence that we could install it safely, and even have potential to move up from 10G later as switches get cheaper. Altogether we pulled about 74 Ethernet and 12 fiber lines.
I also included dual fiber, dual Cat6a, and dual Coax from the current "utility" space where the cable and fiber services enter the house to the new data center. I hope to move those utilities as we finish remodeling the house, but that has yet to be determined. Until then I can bring the incoming cable, fiber, attic antenna to data center - over fiber or Ethernet. I plan to rack mount my Tivo base units, with a Cable as well as Over-the-air box feeding 4k Tivo Minis at each TV. My wife and I had cut the cord, but my in-laws aren't ready to do that yet. I also plan to rack mount my Sonos Connect Amps for all of my basement spaces with these handy 3U shelves, part #262-2967.
In terms of users of the network, part of getting this house was moving my in-laws in so they could have some support as they age, and have single-floor living. My wife and I are remodeling the basement for our master suite. The existing house has wired sound, with Sonos users, and it's a large rambling house with a pool and cabana, so I have dual Ubiquiti HD wireless APs on each end of the main house, 1 in the garage and pool cabana, and 1 each upstairs and in the basement. There will be 1 as well in the new garage we will be adding.
I have installed a bunch of HomeKit automation gear. I have installed a bunch of Leviton Decora Smart Switches to control the house lighting, the fountain, Eve motion detectors and door/window modules, degree temperature sensors, iDevices outdoor switches for outdoor lighting, 5 Nest thermostats, 3 August locks with WiFi extenders, a Rain Bird WiFi sprinkler module, 5 Sonos Connect Amps (and I plan to add several more for the basement remodel), 2 laser printers, 2 multi-function printer, copier, scanners, 5 Ring flood light cameras and 2 Ring doorbell cameras, and about a dozen Ubiquiti cameras (some POE and some micros on WiFi). I have 2 work laptops in the house, 2 personal laptops, 2 desktops, 1 windows server, the UAS, the QNAP NAS, and I plan to add some servers, maybe Dell 720s. I have about 10 Tivo devices, and nearly as many Apple TVs (we do a lot of Airplaying) as well as a HomePod, 4 iPhones, 5 Echos, and 4 Kindles and about 6 "Smart" TVs. Also the Security system will be upgraded to allow WiFi connectivity. At any moment I have nearly 100 WiFi devices.
Nearly as important as having all of this connectivity is keeping it straight, well understood, and maintainable. To that end, I have clear Brother labels on each wall plate with the patch panel number, and each wire has a heat-shrink label with a clear heat shrink protective covering layer to keep it legible. Each cable has a four digit number. The first digit is the patch strip (0/1/2/etc) and then the port number, so "0001" for the first port in first patch panel. The house patch will be connected, by routing back inside the wall, to an inline patch in the respective rack it's served by. The house patch panels are these 24 port Cat6a units - giving me 72 ports for Ethernet, and the wall mount rack is the StarTech 6U "WallMount6". The fiber patch panel is this 12 port unit. For the in-rack connections I'm dropping these SF in-line couplers from the wall patch to feed the switches. I plane to have a patch above and below each 48-port switch to keep the connections nice and tidy.
I'll post my network diagram soon, but here are some pictures midway through construction. As of June 25, 2018 the wire is pulled, tested, and in the jacks. Since there's so much construction, I have a temporary network setup inside the closet, so the patch is kind of nasty looking, with wires poking through a small hole in the rear drywall. Long term I'll clean up this wire, and it will get a OSB or melamine cover that will be painted to match the wall color, and all of the patch cables to the rack will likewise come off the patch and look inside the way, under the flooring, and up into the rack. The floor will be carpeted with low pile commercial carpet squares from FLOR called Lasting Grateness. By the way, all of my 10G NICs for my Macs are Sonnet Twin 10G Thunderbolt 2 units.
As I think I mentioned, I went with a Trane 1.5 ton (18,000 Btu) low-ambient (can cool below freezing outside air temp) mini-split AC unit. This unit can heat, but I shouldn't need that. I will have them mount a hard-wired thermostat on the hot-side wall to verify cooling rather than depend on the wireless handheld unit, which might not be able to see over the 7' tall racks. As you can see, we're framing in for an 8' exterior door to hold the cold in, and provide perimeter security. With the whole-house monitored security and video system this should be adequate.
For the wire pulling I hired Totten wiring services to pull my cable. He's a magician at finding ways to get wire where it needs to go.
Awhile back I bought a Triplett "Real World Certifier" for testing cables. It does a pretty good job, and will show you if you have any breaks or shorts, and verify the wire order and give basic speed capability and an overall "Cat" score on the scale of Cat3-Cat6. It's not a real cable tester, and Jeff will be bringing the real deal over to certify my cables, but it's was a good check before putting some of the lines into temporary service.
Workstation - Monitors for Tivo and serve control
In-Line Ethernet Coupler
Fiber patch panel
Cat6a Patch Panel
6U Wall Mount Rack for patching
Flor "Lasting Grateness" in Indigo
Sonnet Technologies Dual Port 10 Gigabit Ethernet Thunderbolt 2 NIC
2U Rack mount for 3 Sonos Connect Amps
Armored fiber cable
Heat shrink wire label with clear heat shrink cover
Multi-line labels are easier to read from any side
Lots of wire pulled, empty spools in background
Main patch panel is up and running, but messy
Many, many runs completed
New lighting to be seen, the beam soffit will trap hot air on rack side of room
This 8' opening for rolling racks in and out of data center - the inset behind the door is for the ramp, which will get shoe molding to keep you from walking off the edge
Mini split AC compressor
Mini split outside unit mounted above the snow line so it cool data center all winter long
Mini split indoor unit on cold side of data center
Panel
Panel
DSL Reports Speedtest on USG-XG-8 with Full Intrusion Prevention enabled FTTH (Cincinnati Bell Fioptics)
The racks await the room
Circuits mounted in floor await the racks
Labels everywhere to keep things straight
Fiber wall plates
Parts and Bits
Here are the bits and bytes in my Data Center. I apologize that it's a bit rambling, I'll try to make it more coherent later, but this is meant to list the makes, models, etc. of hardware devices in my installation.
Most of the Networking gear is Ubiquiti/UBNT.com stuff running UniFi on a UAS-the UniFi Application Server, or what is now called the UAS-XG Server. It's a Xeon class server with 32GB of ram, and I've replaced the stock video storage to dual 10G Seagate Iron Wolf Pro drives. And the network side it has dual 10G copper ports in a 1U rack mount chassis.
The router is the new UniFi Security Gateway USG-XG-8 with 8 SFP+ ports and 1 10G copper port and 80Gbps routing capability, while doing full IPS/IDS at 1G speeds. It has 16 cores and 16GB of ram in a 1U dual-PSU rack mount chassis. I really only got this so that I could do the IPS/IDS at full 10G like speed between my VLANs. The network is fed by Fiber-to-the-Home/FTTH by Cincinnati Bell Fioptics at 1 Gbps download/250 Mbps upload through an Alcatel-Lucent 7342 ONT, or Optical Network Terminal. I'd love to hook this directly to my router, but they only support GPON, not SFP, and don't support customer equipment for the ONT, though they do allow you to bring your own router.
The core switches are all USW-XG-16 with 12 SFP+ ports and 4 10G copper Ethernet ports. Each rack with have one of these switches connected to the primary by dual 10G multimode fiber patch cables. Each of these switches will support 10G copper links to my servers and 1G or 10G links to my UniFi switches. Currently only the 48 port UniFi switches have 10G SFP+ ports.
My primary distribution switch will be the as-yet-unreleased UniFi 48 port L2 POE switch with dual PSU, the USW-L2-48-POE pair with a USW-48-750W POE switch, giving me 96 ports of POE goodness.
I have various other UniFi switches, including a USW US-16-150W in the rack, various USW-8-150W at the end points (mostly offices) - each of with can accept 1G fiber via SFP, and a bunch of USW-US-8 units, mostly at TVs and gaming consoles. For WiFi I recently swapped out my UAP-AC-Pro units for UAP-AC-HD ones. The HD is slightly bigger than the Pro, but has a lot better throughput. I can get nearly 400 Mbps both directions on my iPhone on the HD, which is about double what I got on the Pros. Also, the beauty of the access points and US-8 switches are that, like the cameras, they're all POE. That means that when my power goes out, these can stay up as each rack is protected with 1 hour of UPS backup. So, as long as I have light on the incoming fiber, I have networking! I currently have 6 of the UAP-AC-HD mounted, 1 on each wing of the first floor, 1 in the garage at ground level, 1 in the basement and 1 on the second floor in the center of the house, and finally one in the pool cabana to cover the backyard entertaining space.
My backup power, and power line conditioning, is by CyberPower OR1500LCDRM1U 1U tall 1,500 VA units in each rack, with dual units in the primary rack as many of those devices have dual PSU. Since these only have 4 protected outlets, I have each powering a Cyber Power CPS1220RMS 1U power strip which support the bulk of power users. Another handy feature of these UPS units, they have an environmental module you can connect to them to monitor temperature in the rack.
My QNAP network attached storage/NAS, a TVS-1271U-RP-i7-32 that I purchased from SPAN.com, will connect directly to the switch with dual 10G fiber. This 2U rackmount NAS has an Intel Core i7 processor and 32GB of ram and 12 SAS/ drive bays, and I've opted for the dual 128MB (IBQ-XRF256) M.2 SSD cache and dual SFP+ expansion (LAN-10G2SF-MLX) cards. I paired this with 12 Seagate Iron Wolf Pro enterprise/NAS-ready 10G SATA (ST10000NE0004) spinning hard drives in a RAID6 array (2 drives for redundancy, so 100TB of storage). On this NAS I run a Plex server with my entire TV show and movie library as well my music and home videos. I also run a VM with a NextCloud instance for all of my personal cloud needs, and use it as an Apple Time Machine target.
I also have a Mac Mini, 2012 model with quad core i7 and 16GB ram running OSX server edition as my web and email server. It has dual 1TB SSD inside the case, and it mounted in a Sonnet 1U "Rack Mini" chassis. It is connected to the network via a Sonnet Twin10G thunderbolt NIC to the copper port on the USW-XG-16. My main workstation is a 5k iMac, also with a 10G Sonnet NIC wired to the USW-XG-16.
The Plex server is mostly used as a host for all of my Tivo gear, mostly the new Tivo Mini 4k VOX units at the TVs, with Tivo Bolt units in the rack. The nice thing about the newer Tivo units is they all have 1G Ethernet ports. The bad thing about the Bolt is the silly up-bend makes it too tall for a 1U rack shelf! I have a cable card version with 6 tuners as well as an over-the-air antenna version. Each TV in the house also has and Apple TV for AirPlay capability and other uses. The primary TVs have either the 4th or 5th generation units, with the balance just being generation 3 models. The 2 newer generations support Plex and other applications, which can be handy.
The entertainment gear, all in rack 3, finishes out with Sonos Connect Amps, in these cool Penn-Elcom rack mounts. Each room will be wired with in-ceiling speakers, a local volume control, and be wired back to Sonos Connect Amp Zone Player. These are cool because they're network-attached amps that can drive the speakers, and can be controlled by pretty much any device on the network. Even better, you can easily group and ungroup multiple zones to play the same content (local, Apple Music, Plex, Air Port Express Airplay content via line-in) or different content. Finally I have a PS4 and a PS3 at different locations for gaming.
For security I have the advertised "Ring of protection", with a Ring doorbell camera and 5 Ring flood light cameras - all on WiFi. Inside of that I have a monitored Honeywell (Stanley Security) wired home security system with door and window sensors, motion and glass breakage, smoke and CO detection. I'm looking to upgrade this with a full LCD touchpad version with cellular and internet connectivity with iOS app support. I also have wired/POE UBNT cameras monitoring all of the indoor and outdoor spaces and recording to the UAS. I have a mix of the new UAP-G3-Pro and older UVC-G3 cameras on the outside and UVC-Micros covering the indoor spaces on WiFi. The Pro is waterproof and has optical zoom, so it's the workhorse, with covered areas and doorways getting the regular G3 with IR extender.
For other home automation tasks, I have a HomePod and several Alexas (although I may get rid of these soon). I have the Rain Bird WiFi Module for my sprinklers, 5 Nest smart thermostats (although we're considering going to geothermal, so these might have to change to Ecobee units), 3 August Smart Locks. One thing I like about most of the home automation gear I buy is that it can function in dumb mode as well. For instance, the Nest thermostat is intuitive to use as a local thermostat to my in-laws. The August lock from the either side can function as just a keyed deadbolt lock if power or connectivity is lost. My Leviton light switch connect to standard wire and function standardly without connectivity.
I have a bunch of HomeKit gear as well, and I love telling Siri on my Apple Watch to do things. I tried the Hue bulbs, but didn't like that mode of interaction (having to leave the switch on all of the time). Instead I've moved on to the Leviton Decora Smart Switch, homekit edition. These are basically just regular switches that have had WiFi added to them, so if data and/or automation fails you can still turn on your lights. I'm considering Lutron Serena smart blinds as well. For automation purposes I've paired the light switches with Elgato Eve Degree temperature and Motion sensors, and the Eve button. In the winter my mother-in-law's office gets chilly. She can use the Eve Button to trigger a iDevices smart wall switch to turn on her space heater, or the Eve Degree can do it automatically to maintain the right local temperature. I have iDevices Indoor and outdoor switches. The outdoor switches control the pump to my fountain, the lights illuminating my fountain, the string lights over my pool, and the bug zapper next to it.
UPDATE: I have started upgrading my on site servers with Dell rack mount units. Partly this is driven by my desire to get better performance and mvoe to virutal servers, and partly due to Apple deprecating many of the OSX (now MacOS) Server features like web and email servers. Previously I'd run Plex on my QNAP NAS, and got occasional complaints about its ability to transcode video. Also, I run a NextCloud private cloud instance on the QNAP in a VM, and although it has an i7 and 32GB of ram, the performance has not been great. So, this move will allow me to dedicate significant compute resources affordable to easy-to-manage, separate, virtual machines - mostly running linux.
My current plan is to split the workload into 3 pieces.
- 1U Dell R620 (ProxMox)
- A thin server with fewer disk slots for less data-intensive workloads
- Domain Name Server (Debian)
- Network Time Server (Debian)
- PiHole Anti-Adware Server (Rasbian)
- Dual Home Assistant instances (Hass.IO) Production/Development
- 2U Dell R720
- Thicker "main" server (ProxMox)
- Email Server (SMTP/IMAP Dovecot/RoundCube)
- Web/Apache Server
- MediaWiki Server
- WordPress Server
- TeamViewer Target/JumpBox
- 2U Dell R720xd (FreeNAS)
- Thicker "SAN" server - Storage Area Network
- Remote storage allows Proxmox "live" migration of Virtual Machines between ProxMox Nodes
- 1 Storage Array of 6 drives (36TB) for Live VM data
- 1 Storage Array of 6 drives (72TB) for VM Backups
Network Attached Storage/QNAP TVS-1271u
1U Rack Server Dell R620 with Bezel
1U Rack Server Dell R620 without its Bezel
2U Rack Server Dell R720 with Bezel
2U Rack Server Dell R720 without its Bezel
2U Rack Server Dell R720xd with Bezel
2U Rack Server Dell R720xd without its Bezel
UAS-Running UniFi Controller and UniFi Video Server
UAS
Router-USG-XG-8 10G capable
XG-8
USW-XG-16 core switches, 1 per rack
Setting up a 20G backbone
Multimode OM4 10G/40G LC-LC Patching Cables
USW-L2-48 and USW-48-750W will be my primary distribution switches
Dual PSU 48 Port UniFi Switch front
Dual PSU 48 Port UniFi Switch back
FTTH ONT-Optical Network Terminal
1 Gpbs download via Optical to Ethernet Terminal (single GPON connector)
16 port POE switch, the USW-US-16-150W
WiFi via UAP-AC-HD
WiFi - old UAP-AC-Pro units replaced
150W POE local 8 port switch, the USW-US-8-150W
POE powered local 8 port switch, the USW-US-8
CyberPower OR1500LCDRM1U UPS front
CyberPower OR1500LCDRM1U UPS back
CyberPower CPS1220RMS power strip front
CyberPower CPS1220RMS power strip front
CyberPower RMCARD205 Networking Card
CyberPower ENVIROSENSOR temperature sensor
Triplett Real World Certifier RWC1000
2012 Mac Mini running OSX Server
Sonnet Rack Mini Front
Sonnet Rack Mini Back
Sonnet Twin10G Thunderbolt 1/2 NIC for all Mac Computers
Sonnet Twin10G NIC
5k iMac 32G
QNAP NAS TVS-1271U-RP-i7-32-US rack mount NAS
X
IBQ-XRF256 QNAP dual M.2 256GB SSD Cache card
IBQ-X10GM QNAP dual SFP+ NIC
12 Seagate Iron Wolf Pro NAS-ready 10G drives ST10000NE0004
Tivo Bolt Cable 4k
Tivo Bolt 4k
Tivo Mini Vox 4k
Penn-Elcom R1498/3UK-SONOSZP120 Connect Amp Rack Mount
Empty Sonos Rack Shelf
Sonos Connect Amp front
Sonos Connect Amp front
Tripp Lite 12 port Keystone N062-0120KJ for Audio Terminations
Keystone binding posts for audio
Patch panel nearly complete
Airport Express for line-in audio to Sonos Connect
Generation 5 Apple TV 4k
Generation 4 Apple TV with Siri Remote
Generation 3 Apple TV with silver remote
Ring Doorbell Pro
Ring Floodlight Security Camera
UBNT G3 Micro security camera
UBNT G3 security camera with IR range extender
UBNT G3 Pro security camera
Apple Home Pod
Amazon Alexa
Rain Bird sprinkler controller Wi-Fi module
Nest smart thermostat generation 2
August smart lock
Leviton Decora HomeKit switches
Eve Button HomeKit
Eve Degree HomeKit
Eve Motion HomeKit
iDevices Outdoor Switch Homekit
Outdoor Switch - Fountain
iDevices Indoor Switch HomeKit
The Plan - Yes, I'm an engineer in my day job
Great Lakes Racks GL840S2-2436
In-Line Ethernet Coupler
Cat6a Patch Panel
Fiber patch panel
6U Wall Mount Rack for patching
9 rolls of Belden Cat6a cable ended up getting bought, and mostly used
Belden 10GXS12 Category 6a cable
Spools of cable await installation
I bought pre-terminated runs of OM4 40G capable armored-jacket fiber optic.
Fan Unit Model: 40MAQB18B--331--
Compessor Unit Model: 38MAQB18R--301--
Trane Mini split AC
Mini split outside unit mounted above the snow line so it cool data center all winter long
Mini split indoor unit on cold side of data center
Configuration
-All View
Interior
Exterior
Back of racks
Front of racks
"Cold Aisle" or front of rack near HVAC unit
Entertainment Rack
Server rack
Networking rack
