Beginning a Data Center Project

By Jim Harrison, RCDD

There has been an explosion in data center growth. The need for more data, information, storage, and applications are outperforming many existing data center infrastructures. Also, many organizations have reached their tipping point on their first or second generation data center and it is now time to consider a new data center to meet the organization’s critical IT infrastructure needs.

In healthcare there has been rapid growth moving towards the Electronic Health Records within hospital and health organizations. Many times the focus is on adding the EHR applications, hardware, and storage to an overwhelmed data center. The current data center is usually out of space, out of power and cooling or does not meet the proper Tier level rating, which creates reliability issues. Organizations have had to find temporary spaces to house this critical infrastructure in order to provide the EHR services to the organization. These temporary spaces include building temporary space or renting space in a co-location facility.

So how do you begin planning that critical data center that will meet your organization’s needs for 10 to 15 years? The first step is to develop an overall roadmap or program. A Program consists of four major elements:

• Definition of the overall needs and requirements
• Conceptual design and space plan
• Overall project budget (including construction and capital expenses)
• Overall schedule from these first initial planning steps through construction and then ultimately the relocation process

Load Analysis

Developing the requirements starts with understanding your business model and care delivery model while also understanding your current IT solutions and future IT solutions. Developing a good base line for where you currently are with equipment loads is also critical. This can be accomplished through an assessment of your current operating environment and current load analysis. After your current loads are established, then an analysis can be conducted for projected growth over the next 10 years and beyond.

LEED Planning, Going Green

Early in the planning process, you need to establish if the project will be a LEED certified project and if so, what Level is desired. This decision will affect the conceptual and budget planning, so establish this requirement early in this process.

Hardness and Reliability

This step addresses how critical and reliable does your data center need to be. First, establish if your entire data center infrastructure needs to have the same level of reliability. After you have established this, then establish a Tier level base line for your most critical need and from there establish less critical areas as needed. The reliability of your data center should consider the uptime needed, how it will be maintained, can you schedule any downtimes to do maintenance on your infrastructure or does it need to be concurrently maintainable, and what happens if power, cooling, and telecommunications are lost. These are just a few considerations. You also have to consider risks that may vary from site to site as to the constructible hardness.

Power System Conceptual Planning

Begin discussions with the local utility company to model operating costs and possible rebates that the design may qualify for to offset the first costs and the annual operating costs. Power planning starts with the desired Tier level and the specific site conditions. Start by looking at the power delivery to the data center IS equipment and then power delivery to the cooling infrastructure. Many conceptual designs might be considered before the best solution is developed. The electrical system concept will also be used for space planning and budgeting to move forward. Also, during the concept planning you must consider your total build-out load based upon 10 year growth requirements versus the initial build out of the infrastructure. You should develop the path to reach the 10 year capacity planning scenario, and how to grow the system beyond the 10 year growth plan. We must also consider how to deliver power to the server cabinets. The options may include PDUs, Panelboards, under floor receptacles, overhead receptacles, and overhead bus duct. Again, consider developing the design concept to always deliver flexible power to each server cabinet at various load capacities of 8KW, to 12KW, to 22KW cooling per cabinet.

Cooling System Conceptual Planning

The conceptual cooling design should also consider the overall Tier level and maintainability of the system. Additionally, the efficiencies and flexibility of the system to deliver cooling to exactly where it is required are also considerations. The overall short term capacity planning and how the system will be able to grow to the 10 year and beyond growth plan should also be analyzed. So in seasonal or colder climates, free cooling should be considered to improve the efficiencies of the system and reduce the overall power consumption. Some design considerations should include using energy recovery wheels, rising the chilled water supply temperatures, and rising the supply temperatures and humidity to servers according to new ASHRAE standards.

A careful analysis of the power usage effectiveness (PUE) based upon the projected loads over time should be considered. The PUE is becoming an industry guideline to establish overall facility efficiency. The PUE rating is found by dividing the overall power consumed by the entire facility by the data center IS equipment load. A PUE rating of 2.0 or less is generally recognized as a general efficient facility. As the number is able to be driven closer to 1.0, the efficiency of the facility goes up, thus your maximum savings on power consumption increases. The EPA has just announced that they will be releasing a new energy management efficiency tracking tool in June of 2010, which will allow companies to begin to apply for an Energy Star rating.

Other cooling considerations are humidity when delivering free cooling using low dew point outside air, air delivery method such as CRACs, or air handlers in a lower level floor delivering air to an access floor supply plenum.

Some other leading concepts today that will provide the higher density cooling requirements of server cabinets containing blade chassis totaling 20KW to 30KW of heat load are cold isles containment and hot air containment. These methods can be used to provide the flexibility and accommodate growth over the life of the data center. One approach is to deliver high capacity cooling to the front of the server cabinet which then rejects heat from the server cabinet through ducted chimneys on the rear of the server cabinets. This hot air containment approach allows for a higher temperature of return air to be captured in the return air plenum, thus allowing for a more efficient cooling system.

We are also seeing a migration away from perimeter CRAC’s to central AHU’s which provide for much more precise control of supply air and a greater delta T split across the equipment when using a common return whether it be ducted or ceiling plenum. The design approach to use close coupled cooling, which can be deployed to specific cabinets and added as needed to meet future high density loads may also be considered.

Network Conceptual Planning

The data center’s network is another critical and costly infrastructure that needs careful planning and consideration. Just like if cooling and power are lost access to the servers are lost, so is the fact that if the network goes down access to the servers is also lost. So first, it is critical to make sure that you have primary and secondary diverse communications paths into the data center. Also having the ability to receive service from diverse service providers may also improve your diversity, reliability, and drive down the circuit cost through competition. So this is a key component when selecting an off campus site for your data center.

Inside your data center there are LAN concepts to consider that will affect the reliability, and cost of the infrastructure. The cabling infrastructure and network hardware make up the LAN design. Some of the concepts and decisions that need to occur are:

• The bandwidth of the cabling infrastructure. Standards are in place for 1, 10, 40, and 100 gigabit delivery system
• Fiber optic cabling vs. copper cabling – type, size, and quantities
• Physical location of the cabling infrastructure and space planning for cable trays that will be needed to support the cabling
• Distributed or centralized network gear or a combination of both. For example will every two server cabinets have a primary and secondary connected network switch?
• Reliability should be address through diverse redundancy of network hardware and the cabling infrastructure
• SAN storage connectivity via Fiber Channel or Ethernet. Centralized or decentralized network gear and how the cabling infrastructure will be designed for reliability and flexibility
• Flexibility of the network to add and remove servers
• How will your KVM be wired?
• Will your back-up principles affect the wiring infrastructure?
• Space for racks and cabinets and the location within the data center

Space Requirements

After electrical, cooling and network concepts have been considered and planned, then the space planning can begin.

Space planning involves determining how much space will be required in two steps. The first analysis involves the space needed for the servers, storage, and data processing equipment. A critical component that is often overlooked is the width of the server cabinet.

The traditional width of server cabinets has been 24”. But most data center managers are moving away from this to a much wider cabinet of 30 to 32 inches. The wider cabinet provides more space for power, network, and SAN cabling. Larger width cabinets provides a higher volume of heat rejections when chimneys are used, and also allow for servers to be removed and added easily without the risk of disrupting the connectivity of the online existing servers in the cabinet.

So within the data center machine room a conceptual plan can be developed based upon some guiding principles that should be established. The goal of the guiding principles should allow for a more efficient use of space, and yield a flexible machine room space where density and flexibility can both be achieved.

Some of the conceptual layout guiding principles may include:

• Size of a typical server cabinet
• Cooling delivery concept
• Power delivery concept
• Density of servers in a cabinet
• Server functionality
• Storage delivery concept
• Initial build out
• Growth strategy
• Service and maintenance working clearances

It also means determining the space requirements for all functional support spaces for the data center such as staff or office space, server build, testing or lab space, service entrance space, command center or NOC, storage space, a loading dock, conference rooms, a receptionist, etc.

Step two involves the sizing and relationship planning space for all infrastructure support spaces after the cooling and electrical conceptual design has been completed. These spaces include, electrical room(s), chiller space, generator space(s), mechanical room(s) fire protect equipment space, and air handler, dry coolers, or CRAC space.

After the space concept decisions have been completed, then a conceptual layout of the entire facility can be developed. The concept should include the machine room layout and all the necessary support spaces. Some other considerations when laying out the conceptual plan include:

• Space relationship
• Constructability
• Column spacing in the machine room
• Roof and exterior wall hardness
• Tier level and space separation
• Flexible and modular growth
• Long term grow strategy beyond 10 years
• LEED certification level if required
• Ceiling and structure heights
• Access floor height if used
• One level, two levels or more

After the conceptual layout is completed, an overall building size and square footage of space can be calculated and used as input in selecting the site.

Site Selection

So at this point we should have design concepts for the power, cooling, and network completed, and have an overall conceptual floor plan and machine room completed.

Site selection is critical. There are many considerations and this is an area where many owners don’t spend enough time making the right choice. They tend to pick an available site and then try to site adapt the requirements or even alter the requirements to fit the site. Many times this leads to higher initial costs or enhances the risk by altering or reducing the requirements.

Once these design concepts have been considered and decided upon, then site criteria can be developed. The criteria will provide valuable information in selecting a site, and will also allow for budget development. Some site selection criteria may include:

• Building or space size
• Building or space for expansion to meet the long term growth plan
• Parking for staff and visitors
• Fuel delivery to the generators if they are diesel
• External risk contributors that may include such factors as weather, earthquake, or proximity to transportation modalities
• Tier level requirements
• Electrical service requirements
• Telecommunication/WAN/LAN requirements
• Site development costs

Relocation Strategy

Another overall planning consideration that should be included in the program and the budget is an overall relocation strategy plan. Many times this piece of the plan is left out and is not considered until the construction is underway. The result of this may affect the overall project schedule and may also lead to overall budget surprises. The relocation budget may typically run between 10 to 40 percent of the overall project budget, depending upon the size and complexity of the relocation. Some early planning considerations include:

• How many various locations will equipment come from?
• What are the relocation constraints?
• How much downtime can be afforded based upon the business or care delivery?
• Are there current or future plans to virtualize servers?
• What are the dependencies and interfaces between the applications?
• How can the system be divided so that relocation can occur in multiple moves?
• Is there vendor supported equipment that will need to be relocated by vendors?
• Can you build a network between the existing and new location for data replication?
• What type and size of bandwidth is required for the network connection?
• Are there new servers, main frame, or storage needed for data replication to reduce risk or reduce downtime?
• Who will move the equipment and in general, how moves are anticipated?

Once these questions are addressed, then a strategic plan can be developed and a budget can be estimated.

Budget

After all the requirements, conceptual planning, and the relocation strategies are complete, then an overall project budget can be developed.

The overall project budget should include both construction and capital expense items for a total comprehensive budget. The budget should include:

• Construction Expense items
  • General construction conditions
  • Site improvements and parking
  • Utilities construction
  • Exterior building and/or interior building construction
  • Electrical (power and lighting) systems and power monitoring
  • Cooling system
  • BAS/Control and monitoring system
  • Fire detection system
  • Fire protection system ( pre-action dry pipe/chemical suppression)
  • Security (access control and video surveillance)
  • Server cabinets
  • Cabling system and support system (copper and fiber optic)
  • Audio visual system(s)
  • Architectural and engineering fees
  • Program management fees
  • System testing prior to commissioning
• Capital Expenses Items
  • LAN, WLAN, and WAN equipment
  • Voice communication system
  • Virtualization project
  • Replication or seed servers, main frame, or storage equipment
  • Relocation/mover(s)
  • Vendor support
  • Commissioning
  • Tier level certification
  • Temporary WAN circuit(s)
  • Relocation insurance

Schedule

To complete the overall programming planning phase an overall schedule should be developed. The schedule should include the planning phase, design phase, construction phase, commissioning phase, network and equipment installation phase, owner testing, and the relocation phases. Today, some of the long lead items that typically run between 16 to 26 weeks to procure are chillers, generators, and intelligent paralleling switchboards.

Conclusion

Completing this overall comprehensive initial planning and programming phase provides management and decision makers with the information that in needed to effectively move forward with a new data center project. This step also provides the architect, engineers, designers, and the construction team with the information that is needed to take the project forward to the next step which is the design phase.

Jim Harrison, RCDD, is a principal, project manager, and serves as Atlanta’s Managing Director. His expertise focus is information technology systems and data centers. He can be reached at jharrison@ediltd.com.