Anatomy of a Datacenter

by Josh Ewin

As designers and developers, our scope of work and our field of view rarely go beyond our screen and the client's website. Seldom do we think about our hosting, except from when installing a script or the server starts lagging. The mechanism that makes our jobs possible in the first place needs to be studied carefully; not only because our livelihoods depend on it, but also because... it is amazing in its design.

Datacenter operations have evolved over the past two decades and now the design and operation of these facilities have become highly standardized. The land rush of the 90s, which fueled the rise of the Internet as we know it, was supported by the expansion of network communications and infrastructure like the hosting datacenters that dot the world and the fiber optics that connect them to the Internet.

The Nuts and Bolts

Datacenters vary widely in size from one small room to a multi-floor facility. But regardless of size, planned design is paramount to successful operations. Datacenters need to be built with white space and available annexation areas for growth, access to network providers and ample power.

Design standards were set to create efficiencies in operation of hosting facilities; however they also assist in defining the caliber and reliability of a datacenter. "Tiering" is a method to grade datacenters according to their design and redundancy. Below are the four specific tiers of datacenters:

Tier 1 - 99.671% uptime: Tier 1 datacenters are basic computer rooms that require a full shutdown for preventative maintenance. Annual downtime is 28.8 hours.
Tier 2 - 99.741% uptime: Tier 2 facilities have some redundancy but still have a single path for power, requiring a shutdown for preventative maintenance. Annual downtime is 22 hours.
Tier 3 - 99.982% uptime: Tier 3 datacenters have sufficient redundancy in place to allow for planned maintenance without downtime (N+1 redundancy.) Tier 3 datacenters are required to have at least 13.2 KV (kilo volts) of power. Annual downtime is 1.6 hours.
Tier 4 - 99.995% uptime: Tier 4 DCs are the top tier. They are built with multiple paths to power and AC and are designed to cope with a worst-case scenario with no critical impact. Tier 4 datacenters are required to have at least 26.2 KV of power. Annual downtime is 0.4 hours.

Today's DCs are designed with redundancy, security, disaster recovery and expansion in mind. If the power goes out, redundant power or battery backup and a diesel generator should kick in. If a network provider goes down or fiber is cut, redundant connectivity needs to pick up the slack. If a piece of equipment fails and catches fire, fire suppression needs to be designed to quickly eliminate the threat and minimize the damage involved.

Datacenter security covers both physical security as well as network security. Closed circuit TV and keycard access set the minimum for datacenter security. Many facilities also employ biometric access, log facility access, and catalog equipment for entering and exiting the facility.

There's good reason for these high security measures - a single piece of networking equipment within a datacenter can cost more than $100,000. Add that to the computer hardware housed within the network operations center (NOC, synonymous with datacenter) and you've just begun to scratch the surface of the overall value held within a facility. But beyond the monetary value of equipment is something priceless - the data. Protecting that data and equipment from threats like fire requires elaborate extinguishing systems. These systems are designed to detect fire early, at the first sign of smoke particles in the air and extinguish the fire with as little collateral damage as possible.

Network Infrastructure

Every datacenter, in order to provide connectivity for its clients, must connect its internal network with the rest of the Internet. Just as there is a difference in the redundancy and security of datacenters, network infrastructure between facilities varies significantly.

Why is network infrastructure important? For starters, visitors to your website probably come from all corners of the earth - that means they will be coming from different networks. More network providers peering at your datacenter means shorter routes between your visitors and your data, and faster connectivity.

Like datacenter tiering, networks are tiered as well. Network tiering, however, is dependent on peering and transit. Larger networks, like Verio, Sprint and Global Crossing are known as Tier 1 networks. Smaller networks, such as local ISPs and some datacenters are known as Tier 4 networks.

Tier 4 - most data centers are tier 4 networks. They have their own internal network but they pay other networks for IP transit outside of the facility.
Tier 3 - tier 3 networks are regional providers. Larger datacenters are typically tier 3 networks and build redundancy through redundant POPs (points of presence) outside of their facilities. Past the POPs, these providers pay for IP transit.
Tier 2 - tier 2 networks have a national or international footprint, but still pay IP transit to reach some portions of the Internet.
Tier 1 - tier 1 providers do not pay IP transit, meaning they have a global presence and don't pay other providers for any portion of their connectivity.

A datacenter's connectivity, its network, needs to be redundant. At the POP or NOC level, a datacenter connects with its network peers through core routers. Redundant networks will carry multiple core routers to ensure connectivity during maintenance periods and in the event of broad network outages. Should one router experience hardware failure, a redundant router is available to route the traffic to the appropriate switch and ultimately to your website.

Staffing

To ensure maximum uptime, constant connectivity and support for equipment and management of any infrastructure issues, you need people. As the industry has matured, standardization, competition and years of price wars have essentially commoditized the hosting industry. These days, what sets competitors apart, more than their network or hardware, is staffing.

A top-tier hosting facility needs to have 24/7 onsite support, no exceptions. If a hard drive in your server fails at 2 a.m., you need someone there to swap it out for a fresh one - someone with experience using the hardware you happen to need. In addition to the folks in the trenches, a full administrative and management staff to support overall operations and planned company expansion is needed.

Choosing a Datacenter

Selection of a datacenter should be a well thought out process; due diligence up front can save you many precious man-hours and ensure that you find a facility and pricing structure that best fits your organization's needs.

While your datacenter doesn't need to be in your backyard, physical proximity is helpful. When trusting your company's online presence to a particular facility, you'll probably want to see their operations. Ask yourself: Is the facility well staffed? Is it free of clutter and are the racks organized and well maintained? If you are shopping locally, any datacenter in your area should welcome you with open arms for a tour.

One thing to keep in mind with regard to hosting - and this is relatively simple - is that not all hosts have datacenters. A large part of the hosting market is reseller hosting. Essentially, value-added resellers resell and provide support to the hosted website. This doesn't mean that your experience with a reseller will be unfruitful. Many resellers add significant value for customers with hosting support and add-on services like domain names and scripts. However, as your company grows you'll need to start dealing directly with a datacenter at some point to save cost and gain control.

So, as you shut down your PC for the night, as you upload that file to the server; think about the complexity, capital expense and the dedication that support the massive
global infrastructure needed to make your website accessible.