Update Your Fire Protection Strategy

By Gary Cudmore

You have just completed a new data center construction project, which took two years to plan and build and cost millions of dollars. The project has been a collaboration of the IT department, facilities and many outside resources to engineer and construct. During this process, you reviewed numerous options for a fire detection and suppression strategy. It was determined that since the facility is manned continuously, it would make sense to invest in the additional expense of an air sampling system. The higher sensitivity of detection allows it to detect a low energy fire in a high air flow environment such as a data center. This is an excellent choice as a first line of defense as it buys additional time before any products of combustion are released, which damage computer components. You also installed a gaseous agent as a second line of defense to extinguish a fire should one not be contained during the incipient stage. This system is tied to the emergency power off system (EPO) and will shunt trip all power to the data center when activated.

The final component of a double interlocked per-action sprinkler system has been added to protect the building structure should the gaseous agent system fail to extinguish the fire. All of these systems have been tested by the fire department to ensure they are operating to design and national and local fire codes.

At your request, you have had the fire protection systems contractors provide system training to ensure the proper response to an emergency.

Now fast forward two years and review your current fire protection strategy and best practices.

By this time, new servers have been removed and added and most likely the electrical and mechanical infrastructure was increased. This in turn has required more connectivity cabling and you may have migrated from copper to fiber or are using both mediums. You may have violated the integrity of the data center envelope and the fire protection systems themselves.

At this moment in time, your data center may be vulnerable to an outage or a total loss of availability.

As you added new servers and grew your support infrastructure, you required numerous outside contractors to perform work in your data center. Do you have established contractor protocol for working in your data center? Do you use integrated methods of procedure (MOPS) to ensure the work is coordinated between different contractors? Which one is responsible for fire proofing any and all penetrations in the data center envelope? This small detail can be a contributing factor in fire protection systems not operating as they were designed. The gaseous agent has a design concentration that must be maintained for the agent to extinguish the fire.

Look around; has a corner become a convenient storage area for boxes, cables, etc.? These are now fuel to a fire incident and will contribute to its propagation. Have you added racks that are taller than the original ones and are they next to a discharge nozzle? Have you reconfigured the layout and added partitions or new walls within the data center?

How often do you test your fire detection and suppression system? Do you test all components of the system to ensure the total system works as designed? Many times the contractor will do an annual system test, which is convenient for them but does not test all components.

Have you been training new data center operators, security personnel and the facility department on the operation of the systems and proper response? Chances are there are many new faces and little or no training has taken place on these systems.

Is it time to update your fire protection strategy and best practices? The answer is: yes, and as soon as possible.

The weakest link in safeguarding the availability of a data center is people. You are relying on many people to ensure your data center is protected; 80 percent of all data center outages are due to human intervention.

Here are some recommended next steps to consider as you update your fire protection strategy and best practices:

• Conduct an internal review with your IT and facilities departments to ensure when contractors are working in your data center they are given detailed instructions on code of conduct. Develop integrated MOPS to coordinate all work between trades to ensure a complete installation and project close-out.
• Review the data center envelope to verify all penetrations are fire proofed to code. Do not take this lightly, just because a contractor put fire putty in a penetration does not mean it was installed properly. No fire proofing or improperly installed fire proofing is commonplace.
• Meet with all fire protection contractors together and review the maintenance programs to ensure all systems are tested as a whole system and on a regular schedule.
• Remove all cabling, power cables, etc., that are abandoned and not in use. This seems obvious, but many data center operators are not good house-keepers.
• Review all emergency signage and placards. Do you have clear instructions in the appropriate areas to direct personnel on the proper next steps during an event? It is human nature to run away from a fire alarm. Will employees think clearly and take decisive action on their own?
• Consider having the fire department perform a site review of your data center. Request they conduct a class on proper use of the handheld fire extinguishers that are wall-mounted in the data center.

Training is the most important step in averting a data center outage or total loss.

• Consider having a training video made specific to your fire systems. This can be used for all new hires as well as a periodic refresher for everyone who has contact with data center operations.
• Consider a dry run simulation with fire protection contractors and data center personnel. Let them hear the alarm bell and horn, see the strobes flashing and simulate the next steps as you review the placards and instruction. This noisy environment will be challenging; just think clearly and communicate with each other. In the end, it will come down to people. How will employees respond? They will be prepared to perform properly in an emergency situation with the proper training and support.

Gary Cudmore has been engineering and constructing data centers and technical environments for more than 25 years.  He serves as Principal for Data Center Services at EDI. He can be reached at gcudmore@ediltd.com.